[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:01 dist-test-slave-s4p1 krb5kdc[5189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:01 dist-test-slave-s4p1 krb5kdc[5189](info): set up 2 sockets
Sep 26 14:50:01 dist-test-slave-s4p1 krb5kdc[5189](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 14:50:02.400513 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:02.517601 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41393
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:41393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:02.595494 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:02.595690 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:02.595711 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:02.596859 5202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:02.596900 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:02.596915 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:02.596966 5202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:02.596994 5202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:02.598486 5202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:41393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:02.598738 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:02.598991 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:02.604063 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:02.604130 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:02.604120 5202 server_base.cc:1034] running on GCE node
W20260926 14:50:02.604082 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:02.604636 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:02.604935 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:02.606098 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790434202606094 us; error 35 us; skew 500 ppm
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:02.608625 5202 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:02.608992 5202 webserver.cc:460] Webserver started at http://127.5.15.190:44689/ using document root <none> and password file <none>
I20260926 14:50:02.609180 5202 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:02.609220 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:02.609305 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:02.610282 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "f307db1deaef43f2a911a8a2b5114188"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "50c706c42bf541f640a051cd6accda22"
server_key_iv: "25f7b78f53d9323c6ee5284d08663a4f"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.610595 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "f307db1deaef43f2a911a8a2b5114188"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "50c706c42bf541f640a051cd6accda22"
server_key_iv: "25f7b78f53d9323c6ee5284d08663a4f"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.611899 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:02.612560 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.612708 5202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:02.612792 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "f307db1deaef43f2a911a8a2b5114188"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "50c706c42bf541f640a051cd6accda22"
server_key_iv: "25f7b78f53d9323c6ee5284d08663a4f"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.612843 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:02.627995 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:02.629685 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:02.629786 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:02.633533 5202 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:41393
I20260926 14:50:02.633601 5269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:41393 every 8 connection(s)
I20260926 14:50:02.633940 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:02.634784 5270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:02.637006 5270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188: Bootstrap starting.
I20260926 14:50:02.637583 5270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:02.637868 5270 log.cc:824] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:02.638522 5270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188: No bootstrap required, opened a new log
I20260926 14:50:02.640092 5270 raft_consensus.cc:348] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f307db1deaef43f2a911a8a2b5114188" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41393 } }
I20260926 14:50:02.640250 5270 raft_consensus.cc:374] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:02.640297 5270 raft_consensus.cc:729] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f307db1deaef43f2a911a8a2b5114188, State: Initialized, Role: FOLLOWER
I20260926 14:50:02.640456 5270 consensus_queue.cc:260] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f307db1deaef43f2a911a8a2b5114188" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41393 } }
I20260926 14:50:02.640555 5270 raft_consensus.cc:388] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:02.640600 5270 raft_consensus.cc:482] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:02.640650 5270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:02.641147 5270 raft_consensus.cc:504] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f307db1deaef43f2a911a8a2b5114188" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41393 } }
I20260926 14:50:02.641260 5270 leader_election.cc:302] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f307db1deaef43f2a911a8a2b5114188; no voters:
I20260926 14:50:02.641430 5270 leader_election.cc:288] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:02.641465 5273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:02.641634 5273 raft_consensus.cc:686] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [term 1 LEADER]: Becoming Leader. State: Replica: f307db1deaef43f2a911a8a2b5114188, State: Running, Role: LEADER
I20260926 14:50:02.641686 5270 sys_catalog.cc:564] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:02.641719 5273 consensus_queue.cc:237] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f307db1deaef43f2a911a8a2b5114188" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41393 } }
I20260926 14:50:02.642289 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5202
I20260926 14:50:02.642398 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:02.643042 5275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f307db1deaef43f2a911a8a2b5114188. Latest consensus state: current_term: 1 leader_uuid: "f307db1deaef43f2a911a8a2b5114188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f307db1deaef43f2a911a8a2b5114188" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41393 } } }
I20260926 14:50:02.643139 5275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:02.643208 5274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f307db1deaef43f2a911a8a2b5114188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f307db1deaef43f2a911a8a2b5114188" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41393 } } }
I20260926 14:50:02.643239 5182 external_mini_cluster.cc:1428] Setting key 7aed2cee01df6bdc6a8a7be740e6f008
I20260926 14:50:02.643275 5274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:02.643344 5289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:02.644023 5289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:02.645592 5289 catalog_manager.cc:1347] Generated new cluster ID: b5d74cbd1d6640c2b47d4670eaffcd7e
I20260926 14:50:02.645633 5289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:02.661772 5289 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:02.662458 5289 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:02.664002 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:02.644163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.5.15.190:41393)
0926 14:50:02.646861 (+ 2698us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:02.646869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:02.647117 (+ 248us) server_negotiation.cc:407] Connection header received
0926 14:50:02.649374 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:02.649381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:02.649450 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:02.649573 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:02.651661 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.653376 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.654900 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.655472 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.655573 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:02.659770 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:02.659775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:02.659775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:02.659787 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:02.661978 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:02.662476 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:02.662479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:02.662480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:02.662591 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:02.662915 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:02.662916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:02.662917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:02.663191 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:02.663351 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:02.663623 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 14:50:02.663771 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2649,"thread_start_us":81,"threads_started":1}
I20260926 14:50:02.667857 5289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f307db1deaef43f2a911a8a2b5114188: Generated new TSK 0
I20260926 14:50:02.668005 5289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:02.699266 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41393
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:02.791388 5296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:02.791599 5296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:02.791635 5296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:02.792729 5296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:02.792788 5296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:02.792830 5296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:02.794236 5296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41393
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:02.794495 5296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:02.794719 5296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:02.799502 5302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:02.799561 5296 server_base.cc:1034] running on GCE node
W20260926 14:50:02.799515 5304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:02.799515 5301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:02.799923 5296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:02.800174 5296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:02.801333 5296 hybrid_clock.cc:648] HybridClock initialized: now 1790434202801315 us; error 36 us; skew 500 ppm
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:02.806993 5296 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:02.807358 5296 webserver.cc:460] Webserver started at http://127.5.15.129:42917/ using document root <none> and password file <none>
I20260926 14:50:02.807539 5296 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:02.807577 5296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:02.807662 5296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:02.808599 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "4ecbbf49a2854ef6af5c6d75eb262e59"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "756ade458917c380469eae20ba958177"
server_key_iv: "c7532459639de9776cb373a19e147b79"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.808890 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "4ecbbf49a2854ef6af5c6d75eb262e59"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "756ade458917c380469eae20ba958177"
server_key_iv: "c7532459639de9776cb373a19e147b79"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.810262 5296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:02.810920 5311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.811079 5296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:02.811157 5296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "4ecbbf49a2854ef6af5c6d75eb262e59"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "756ade458917c380469eae20ba958177"
server_key_iv: "c7532459639de9776cb373a19e147b79"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.811208 5296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:02.823462 5296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:02.825696 5296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:02.825802 5296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:02.825976 5296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:02.826279 5296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:02.826313 5296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.826335 5296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:02.826381 5296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.831707 5296 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:39637
I20260926 14:50:02.831776 5424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:39637 every 8 connection(s)
I20260926 14:50:02.832067 5296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:02.834661 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5296
I20260926 14:50:02.834733 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:02.834923 5182 external_mini_cluster.cc:1428] Setting key 5f40f46fa33de9aa6cb4840a90bfab5d
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:02.842157 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:02.832798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:54927 (local address 127.5.15.190:41393)
0926 14:50:02.832848 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:02.832851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:02.833067 (+ 216us) server_negotiation.cc:407] Connection header received
0926 14:50:02.833509 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:02.833509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:02.833519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:02.833571 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:02.834464 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.835147 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.836612 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.836843 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.836927 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:02.839354 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:02.839356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:02.839356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:02.839364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:02.840759 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:02.841238 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:02.841239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:02.841239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:02.841285 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:02.841542 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:02.841542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:02.841543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:02.841638 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:02.841710 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:02.842029 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 14:50:02.842096 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:02.842474 5427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:02.832791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41393 (local address 127.5.15.129:54927)
0926 14:50:02.832950 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:02.832984 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:02.833428 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:02.833624 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:02.833627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:02.834206 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:02.834391 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:02.834395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.835240 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.835241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:02.836531 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:02.836534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.836948 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.836950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:02.837077 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:02.837559 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:02.837567 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:02.839237 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:02.840842 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:02.840850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:02.840852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:02.841158 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:02.841347 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:02.841347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:02.841348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:02.841480 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:02.841721 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:02.841726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:02.841887 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 14:50:02.842008 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 14:50:02.842178 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260926 14:50:02.843031 5425 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41393
I20260926 14:50:02.843154 5425 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:02.843356 5425 heartbeater.cc:511] Master 127.5.15.190:41393 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
I20260926 14:50:02.843941 5234 ts_manager.cc:194] Registered new tserver with Master: 4ecbbf49a2854ef6af5c6d75eb262e59 (127.5.15.129:39637)
I20260926 14:50:02.844533 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:54927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:02.859330 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41393
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:02.950560 5430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:02.950788 5430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:02.950824 5430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:02.951948 5430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:02.951992 5430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:02.952044 5430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:02.953467 5430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41393
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:02.953688 5430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:02.953897 5430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:02.958595 5435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:02.958580 5438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:02.958799 5430 server_base.cc:1034] running on GCE node
W20260926 14:50:02.958576 5436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:02.959043 5430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:02.959283 5430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:02.960420 5430 hybrid_clock.cc:648] HybridClock initialized: now 1790434202960414 us; error 29 us; skew 500 ppm
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:02.962899 5430 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:02.963373 5430 webserver.cc:460] Webserver started at http://127.5.15.130:34015/ using document root <none> and password file <none>
I20260926 14:50:02.963586 5430 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:02.963629 5430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:02.963750 5430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:02.964725 5430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "97ad439bd21246dc858c6a13770c40f8"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "3794f3b9c903b503da7e67baae750103"
server_key_iv: "4074649c1fbb518f7c70c981d85cbb30"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.965080 5430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "97ad439bd21246dc858c6a13770c40f8"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "3794f3b9c903b503da7e67baae750103"
server_key_iv: "4074649c1fbb518f7c70c981d85cbb30"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.966379 5430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:02.967128 5445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.967303 5430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.967383 5430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "97ad439bd21246dc858c6a13770c40f8"
format_stamp: "Formatted at 2026-09-26 14:50:02 on dist-test-slave-s4p1"
server_key: "3794f3b9c903b503da7e67baae750103"
server_key_iv: "4074649c1fbb518f7c70c981d85cbb30"
server_key_version: "encryptionkey@0"
I20260926 14:50:02.967469 5430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:02.986147 5430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:02.987928 5430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:02.988071 5430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:02.988268 5430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:02.988557 5430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:02.988610 5430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.988646 5430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:02.988673 5430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:02.994189 5430 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:35445
I20260926 14:50:02.994225 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:35445 every 8 connection(s)
I20260926 14:50:02.994541 5430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:02 dist-test-slave-s4p1 krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434202, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:03.002718 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:02.995230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:35291 (local address 127.5.15.190:41393)
0926 14:50:02.995269 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:02.995271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:02.995450 (+ 179us) server_negotiation.cc:407] Connection header received
0926 14:50:02.995889 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:02.995890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:02.995898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:02.995952 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:02.996835 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.997321 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.998198 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.998363 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.998409 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:03.000480 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:03.000481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:03.000482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:03.000489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:03.001646 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.002008 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.002009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.002009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.002054 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.002248 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.002248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.002249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.002340 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:03.002387 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:03.002602 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 14:50:03.002658 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:50:03.002954 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:02.995180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41393 (local address 127.5.15.130:35291)
0926 14:50:02.995334 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:02.995373 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:02.995810 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:02.996030 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:02.996033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:02.996585 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:02.996754 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:02.996758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.997395 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.997396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:02.998132 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:02.998134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:02.998427 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:02.998428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:02.998516 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:02.999090 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:02.999098 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:03.000392 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:03.001724 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:03.001730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:03.001731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:03.001943 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:03.002109 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:03.002110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:03.002110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:03.002196 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:03.002392 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:03.002395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:03.002496 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 14:50:03.002577 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:50:03.002682 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260926 14:50:03.003419 5559 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41393
I20260926 14:50:03.003486 5559 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:03.003610 5559 heartbeater.cc:511] Master 127.5.15.190:41393 requested a full tablet report, sending...
I20260926 14:50:03.003952 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5430
I20260926 14:50:03.003983 5234 ts_manager.cc:194] Registered new tserver with Master: 97ad439bd21246dc858c6a13770c40f8 (127.5.15.130:35445)
I20260926 14:50:03.004050 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:03.004207 5182 external_mini_cluster.cc:1428] Setting key 1dbed993e3299f29f0544d90845f2b29
I20260926 14:50:03.004453 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:35291
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:03.029956 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41393
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:03.107726 5564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:03.107911 5564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:03.107933 5564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:03.109035 5564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:03.109074 5564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:03.109123 5564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:03.110481 5564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41393
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:03.110694 5564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:03.110918 5564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:03.115495 5572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:03.115502 5569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:03.115502 5570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:03.115819 5564 server_base.cc:1034] running on GCE node
I20260926 14:50:03.116052 5564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:03.116288 5564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:03.117452 5564 hybrid_clock.cc:648] HybridClock initialized: now 1790434203117432 us; error 37 us; skew 500 ppm
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:03.120154 5564 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:03.120558 5564 webserver.cc:460] Webserver started at http://127.5.15.131:33563/ using document root <none> and password file <none>
I20260926 14:50:03.120766 5564 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:03.120829 5564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:03.120972 5564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:03.121961 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "482801bf20c7416abcb6f85a4a84565c"
format_stamp: "Formatted at 2026-09-26 14:50:03 on dist-test-slave-s4p1"
server_key: "65f5861ab73d23c8bec1493eff81d4a6"
server_key_iv: "11a1946ad1afe1ad07f81d330f54fadf"
server_key_version: "encryptionkey@0"
I20260926 14:50:03.122299 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "482801bf20c7416abcb6f85a4a84565c"
format_stamp: "Formatted at 2026-09-26 14:50:03 on dist-test-slave-s4p1"
server_key: "65f5861ab73d23c8bec1493eff81d4a6"
server_key_iv: "11a1946ad1afe1ad07f81d330f54fadf"
server_key_version: "encryptionkey@0"
I20260926 14:50:03.123625 5564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:03.124346 5579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:03.124540 5564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:03.124614 5564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "482801bf20c7416abcb6f85a4a84565c"
format_stamp: "Formatted at 2026-09-26 14:50:03 on dist-test-slave-s4p1"
server_key: "65f5861ab73d23c8bec1493eff81d4a6"
server_key_iv: "11a1946ad1afe1ad07f81d330f54fadf"
server_key_version: "encryptionkey@0"
I20260926 14:50:03.124662 5564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:03.134931 5564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:03.136585 5564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:03.136715 5564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:03.136910 5564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:03.137179 5564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:03.137230 5564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:03.137267 5564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:03.137295 5564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:03.142685 5564 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:36439
I20260926 14:50:03.142767 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:36439 every 8 connection(s)
I20260926 14:50:03.143086 5564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:03.144047 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5564
I20260926 14:50:03.144142 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:03.144297 5182 external_mini_cluster.cc:1428] Setting key 4fdfac309d1709e294eb6314d5abfe8c
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434203, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:03.151778 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:03.143800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:60525 (local address 127.5.15.190:41393)
0926 14:50:03.143854 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:03.143856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:03.144068 (+ 212us) server_negotiation.cc:407] Connection header received
0926 14:50:03.144673 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:03.144674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:03.144686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:03.144754 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:03.145834 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.146303 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.147434 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.147604 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.147653 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:03.149512 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:03.149513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:03.149514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:03.149521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:03.150663 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.151083 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.151084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.151084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.151133 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.151316 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.151316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.151317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.151404 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:03.151450 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:03.151661 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 14:50:03.151720 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:03.151991 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:03.143781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41393 (local address 127.5.15.131:60525)
0926 14:50:03.143939 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:03.143976 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:03.144559 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:03.144817 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:03.144821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:03.145565 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:03.145749 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:03.145753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.146508 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.146509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:03.147365 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:03.147367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.147672 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.147673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:03.147758 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:03.148248 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:03.148255 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:03.149433 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:03.150801 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:03.150813 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:03.150814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:03.151019 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:03.151187 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:03.151188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:03.151188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:03.151267 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:03.151452 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:03.151456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:03.151547 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:50:03.151636 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 14:50:03.151739 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260926 14:50:03.152488 5693 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41393
I20260926 14:50:03.152568 5693 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:03.152685 5693 heartbeater.cc:511] Master 127.5.15.190:41393 requested a full tablet report, sending...
I20260926 14:50:03.153003 5233 ts_manager.cc:194] Registered new tserver with Master: 482801bf20c7416abcb6f85a4a84565c (127.5.15.131:36439)
I20260926 14:50:03.153322 5233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:60525
I20260926 14:50:03.156138 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:03.177233 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:03.170241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.5.15.129:39637)
0926 14:50:03.170395 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:03.170399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:03.170409 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:03.170432 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:03.170434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:03.170446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:03.170508 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:03.171106 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.171786 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.172534 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.172826 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.172895 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:03.174998 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:03.175004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:03.175004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:03.175010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:03.176328 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.176629 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.176631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.176631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.176689 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.176831 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.176831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.176832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.176937 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:03.177042 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:03.177101 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 14:50:03.177161 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
W20260926 14:50:03.177582 5339 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47426
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:03.197299 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:03.191005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.5.15.129:39637)
0926 14:50:03.191072 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:03.191074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:03.191136 (+ 62us) server_negotiation.cc:407] Connection header received
0926 14:50:03.191227 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:03.191228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:03.191238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:03.191304 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:03.191805 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.192311 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.193009 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:03.193240 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:03.193321 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:03.195287 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:03.195288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:03.195288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:03.195295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:03.196470 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.196774 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.196775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.196775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.196818 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:03.196964 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:03.196964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:03.196964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:03.197047 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:03.197129 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:03.197191 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:50:03.197243 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:50:03.197773 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5296
I20260926 14:50:03.201921 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5430
I20260926 14:50:03.206943 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5564
I20260926 14:50:03.210927 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5202
2026-09-26T14:50:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1849 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5707](info): set up 2 sockets
Sep 26 14:50:03 dist-test-slave-s4p1 krb5kdc[5707](info): commencing operation
krb5kdc: starting...
W20260926 14:50:04.230106 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:04.347553 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38249
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:04.425904 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:04.426093 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:04.426115 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:04.427301 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:04.427341 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:04.427356 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:04.427368 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:04.427417 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:04.428822 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:04.429025 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:04.429225 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:04.433925 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:04.433971 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:04.434142 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:04.434128 5720 server_base.cc:1034] running on GCE node
I20260926 14:50:04.434401 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:04.434686 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:04.435822 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790434204435819 us; error 36 us; skew 500 ppm
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:04.438256 5720 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:04.438704 5720 webserver.cc:460] Webserver started at http://127.5.15.190:42011/ using document root <none> and password file <none>
I20260926 14:50:04.438975 5720 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:04.439046 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:04.439188 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:04.440193 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "eff3a00a1e57422d845ab2deb464085f"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "d17e8c936d746eee8c75c8bad069ee63"
server_key_iv: "d40843414774c4f7545950ec32bf538f"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.440541 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "eff3a00a1e57422d845ab2deb464085f"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "d17e8c936d746eee8c75c8bad069ee63"
server_key_iv: "d40843414774c4f7545950ec32bf538f"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.441787 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:04.442489 5735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.442656 5720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:04.442752 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "eff3a00a1e57422d845ab2deb464085f"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "d17e8c936d746eee8c75c8bad069ee63"
server_key_iv: "d40843414774c4f7545950ec32bf538f"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.442894 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:04.478668 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:04.481057 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:04.481217 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:04.485239 5720 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:38249
I20260926 14:50:04.485275 5787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:38249 every 8 connection(s)
I20260926 14:50:04.485653 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:04.486519 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:04.488912 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f: Bootstrap starting.
I20260926 14:50:04.489531 5788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:04.489814 5788 log.cc:824] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:04.490463 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f: No bootstrap required, opened a new log
I20260926 14:50:04.492244 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5720
I20260926 14:50:04.492130 5788 raft_consensus.cc:348] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff3a00a1e57422d845ab2deb464085f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38249 } }
I20260926 14:50:04.492295 5788 raft_consensus.cc:374] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:04.492324 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:04.492332 5788 raft_consensus.cc:729] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff3a00a1e57422d845ab2deb464085f, State: Initialized, Role: FOLLOWER
I20260926 14:50:04.492477 5182 external_mini_cluster.cc:1428] Setting key fb54a6b9475e44c4a65fe290fa43c449
I20260926 14:50:04.492522 5788 consensus_queue.cc:260] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff3a00a1e57422d845ab2deb464085f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38249 } }
I20260926 14:50:04.492609 5788 raft_consensus.cc:388] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:04.492642 5788 raft_consensus.cc:482] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:04.492695 5788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:04.493350 5788 raft_consensus.cc:504] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff3a00a1e57422d845ab2deb464085f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38249 } }
I20260926 14:50:04.493472 5788 leader_election.cc:302] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff3a00a1e57422d845ab2deb464085f; no voters:
I20260926 14:50:04.493647 5788 leader_election.cc:288] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:04.493673 5793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:04.493826 5793 raft_consensus.cc:686] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [term 1 LEADER]: Becoming Leader. State: Replica: eff3a00a1e57422d845ab2deb464085f, State: Running, Role: LEADER
I20260926 14:50:04.493884 5788 sys_catalog.cc:564] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:04.493984 5793 consensus_queue.cc:237] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff3a00a1e57422d845ab2deb464085f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38249 } }
I20260926 14:50:04.494441 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eff3a00a1e57422d845ab2deb464085f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff3a00a1e57422d845ab2deb464085f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38249 } } }
I20260926 14:50:04.494521 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:04.494738 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eff3a00a1e57422d845ab2deb464085f. Latest consensus state: current_term: 1 leader_uuid: "eff3a00a1e57422d845ab2deb464085f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff3a00a1e57422d845ab2deb464085f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38249 } } }
I20260926 14:50:04.495108 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:04.495242 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:04.495728 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:04.497247 5801 catalog_manager.cc:1347] Generated new cluster ID: 04dd6bf9cc444d3ebe79379e75735d16
I20260926 14:50:04.497330 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:04.504709 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.492994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.5.15.190:38249)
0926 14:50:04.493434 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:04.493443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:04.493475 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:04.493608 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:04.493614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:04.493671 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:04.493815 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:04.494785 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.496355 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.497316 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.497892 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.498016 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.500677 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:04.500681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:04.500681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:04.500692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:04.503099 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.503511 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.503514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.503514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.503620 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.503824 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.503825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.503825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.504095 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:04.504232 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:04.504359 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:50:04.504488 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":366,"threads_started":1}
I20260926 14:50:04.506347 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:04.507055 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:04.520143 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eff3a00a1e57422d845ab2deb464085f: Generated new TSK 0
I20260926 14:50:04.520323 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:04.546674 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38249
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:04.626037 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:04.626240 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:04.626289 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:04.627477 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:04.627539 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:04.627583 5814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:04.628983 5814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:04.629230 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:04.629436 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:04.634081 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:04.634254 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:04.634156 5814 server_base.cc:1034] running on GCE node
W20260926 14:50:04.634230 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:04.634596 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:04.634877 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:04.636031 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790434204636079 us; error 88 us; skew 500 ppm
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:04.638396 5814 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:04.638792 5814 webserver.cc:460] Webserver started at http://127.5.15.129:40097/ using document root <none> and password file <none>
I20260926 14:50:04.639014 5814 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:04.639055 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:04.639140 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:04.640049 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "2d876da6de214059a832e83f825b7c63"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "ff1c05630a12fdf2355aa481e646cc60"
server_key_iv: "803142eb7497b7d3c053af1b6076baf5"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.640338 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "2d876da6de214059a832e83f825b7c63"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "ff1c05630a12fdf2355aa481e646cc60"
server_key_iv: "803142eb7497b7d3c053af1b6076baf5"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.641578 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:04.642220 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.642362 5814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:04.642459 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "2d876da6de214059a832e83f825b7c63"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "ff1c05630a12fdf2355aa481e646cc60"
server_key_iv: "803142eb7497b7d3c053af1b6076baf5"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.642511 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:04.665298 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:04.667235 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:04.667337 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:04.667514 5814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:04.667820 5814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:04.667852 5814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.667873 5814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:04.667948 5814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.673322 5814 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:41477
I20260926 14:50:04.673393 5942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:41477 every 8 connection(s)
I20260926 14:50:04.673744 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:04.681478 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5814
I20260926 14:50:04.681579 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:04.681756 5182 external_mini_cluster.cc:1428] Setting key d5362f492038d7d81f708eabcc6ce64a
I20260926 14:50:04.683140 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.674496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:34787 (local address 127.5.15.190:38249)
0926 14:50:04.674563 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:04.674565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:04.674729 (+ 164us) server_negotiation.cc:407] Connection header received
0926 14:50:04.675701 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:04.675702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:04.675711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:04.675761 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:04.676314 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.676802 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.677737 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.677926 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.677974 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.680100 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:04.680101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:04.680101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:04.680107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:04.681636 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.682160 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.682163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.682163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.682225 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.682469 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.682470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.682471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.682587 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:04.682660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:04.682970 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 14:50:04.683051 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:04.683336 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.674455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38249 (local address 127.5.15.129:34787)
0926 14:50:04.674609 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:04.674648 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:04.675614 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:04.675823 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:04.675826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:04.676065 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:04.676247 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:04.676252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.676877 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.676878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:04.677670 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:04.677672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.677993 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.677994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:04.678092 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.678600 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:04.678608 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:04.680017 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:04.681717 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:04.681722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:04.681723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:04.682077 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:04.682290 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:04.682291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:04.682292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:04.682394 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:04.682651 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:04.682658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:04.682840 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 14:50:04.682946 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 14:50:04.683098 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260926 14:50:04.683892 5943 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38249
I20260926 14:50:04.683992 5943 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:04.684161 5943 heartbeater.cc:511] Master 127.5.15.190:38249 requested a full tablet report, sending...
I20260926 14:50:04.684697 5752 ts_manager.cc:194] Registered new tserver with Master: 2d876da6de214059a832e83f825b7c63 (127.5.15.129:41477)
I20260926 14:50:04.685617 5752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:34787
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:04.705389 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38249
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:04.782124 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:04.782297 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:04.782320 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:04.783427 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:04.783466 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:04.783492 5948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:04.784916 5948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:04.785140 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:04.785346 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:04.789880 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:04.789950 5948 server_base.cc:1034] running on GCE node
W20260926 14:50:04.790081 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:04.790024 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:04.790295 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:04.790510 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:04.791651 5948 hybrid_clock.cc:648] HybridClock initialized: now 1790434204791640 us; error 31 us; skew 500 ppm
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:04.793921 5948 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:04.794359 5948 webserver.cc:460] Webserver started at http://127.5.15.130:38367/ using document root <none> and password file <none>
I20260926 14:50:04.794603 5948 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:04.794684 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:04.794860 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:04.795862 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "727a6ab0b10f404bbeedaa0b1eeef68e"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "1d4dd66b180bee84f1578861826ce39c"
server_key_iv: "2df7587ede946bca3b93344cbb3b9c6a"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.796208 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "727a6ab0b10f404bbeedaa0b1eeef68e"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "1d4dd66b180bee84f1578861826ce39c"
server_key_iv: "2df7587ede946bca3b93344cbb3b9c6a"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.797521 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:04.798167 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.798321 5948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:04.798408 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "727a6ab0b10f404bbeedaa0b1eeef68e"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "1d4dd66b180bee84f1578861826ce39c"
server_key_iv: "2df7587ede946bca3b93344cbb3b9c6a"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.798477 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:04.811642 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:04.813356 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:04.813509 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:04.813712 5948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:04.813988 5948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:04.814037 5948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.814070 5948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:04.814129 5948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.819923 5948 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:44203
I20260926 14:50:04.819967 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:44203 every 8 connection(s)
I20260926 14:50:04.820264 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:04.828486 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.820949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:42669 (local address 127.5.15.190:38249)
0926 14:50:04.820991 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:04.820993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:04.821214 (+ 221us) server_negotiation.cc:407] Connection header received
0926 14:50:04.822084 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:04.822085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:04.822094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:04.822150 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:04.822695 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.823218 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.824118 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.824280 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.824326 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.826273 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:04.826274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:04.826274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:04.826280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:04.827473 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.827798 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.827798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.827799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.827844 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.828035 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.828035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.828035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.828132 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:04.828178 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:04.828376 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 14:50:04.828428 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:04.828640 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.820910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38249 (local address 127.5.15.130:42669)
0926 14:50:04.821078 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:04.821116 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:04.822002 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:04.822198 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:04.822201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:04.822441 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:04.822627 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:04.822631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.823292 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.823292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:04.824050 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:04.824052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.824340 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.824340 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:04.824454 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.824945 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:04.824952 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:04.826199 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:04.827535 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:04.827540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:04.827541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:04.827732 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:04.827895 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:04.827896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:04.827896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:04.827984 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:04.828181 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:04.828184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:04.828279 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 14:50:04.828352 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:04.828463 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260926 14:50:04.829078 6077 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38249
I20260926 14:50:04.829177 6077 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:04.829334 6077 heartbeater.cc:511] Master 127.5.15.190:38249 requested a full tablet report, sending...
I20260926 14:50:04.829711 5752 ts_manager.cc:194] Registered new tserver with Master: 727a6ab0b10f404bbeedaa0b1eeef68e (127.5.15.130:44203)
I20260926 14:50:04.829747 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 5948
I20260926 14:50:04.829834 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:04.829982 5182 external_mini_cluster.cc:1428] Setting key 3767fc413221c4aedb7da24ba846c9b6
I20260926 14:50:04.830135 5752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:42669
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:04.851604 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38249
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:04.926610 6082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:04.926819 6082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:04.926842 6082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:04.927979 6082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:04.928018 6082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:04.928068 6082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:04.929445 6082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38249
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:04.929684 6082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:04.929925 6082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:04.934476 6088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:04.934541 6090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:04.934463 6087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:04.934787 6082 server_base.cc:1034] running on GCE node
I20260926 14:50:04.935020 6082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:04.935279 6082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:04.936420 6082 hybrid_clock.cc:648] HybridClock initialized: now 1790434204936409 us; error 34 us; skew 500 ppm
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:04.938934 6082 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:04.939321 6082 webserver.cc:460] Webserver started at http://127.5.15.131:34519/ using document root <none> and password file <none>
I20260926 14:50:04.939541 6082 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:04.939608 6082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:04.939736 6082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:04.940706 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "676f91ba096246a1b2d14679e355e510"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "d65ee59eb8ab0e2272675f099e7b1b2b"
server_key_iv: "7104b8027daeeee25f06885659d875bf"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.941061 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "676f91ba096246a1b2d14679e355e510"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "d65ee59eb8ab0e2272675f099e7b1b2b"
server_key_iv: "7104b8027daeeee25f06885659d875bf"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.942250 6082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:04.942994 6097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.943162 6082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:04.943253 6082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "676f91ba096246a1b2d14679e355e510"
format_stamp: "Formatted at 2026-09-26 14:50:04 on dist-test-slave-s4p1"
server_key: "d65ee59eb8ab0e2272675f099e7b1b2b"
server_key_iv: "7104b8027daeeee25f06885659d875bf"
server_key_version: "encryptionkey@0"
I20260926 14:50:04.943333 6082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:04.970894 6082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:04.973127 6082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:04.973282 6082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:04.973508 6082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:04.973803 6082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:04.973855 6082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.973905 6082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:04.973937 6082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:04.979466 6082 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:36111
I20260926 14:50:04.979542 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:36111 every 8 connection(s)
I20260926 14:50:04.979837 6082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:04 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:04.985766 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6082
I20260926 14:50:04.985883 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:04.986053 5182 external_mini_cluster.cc:1428] Setting key fc74cfb492812408584d7523b4513101
I20260926 14:50:04.988988 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.980515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:39897 (local address 127.5.15.190:38249)
0926 14:50:04.980556 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:04.980557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:04.980770 (+ 213us) server_negotiation.cc:407] Connection header received
0926 14:50:04.981586 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:04.981587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:04.981596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:04.981651 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:04.982174 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.982656 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.983610 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.983785 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.983837 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.986292 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:04.986294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:04.986294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:04.986302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:04.987689 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.988091 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.988092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.988092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.988149 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:04.988384 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:04.988385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:04.988385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:04.988500 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:04.988572 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:04.988824 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 14:50:04.988905 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:04.989147 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.980484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38249 (local address 127.5.15.131:39897)
0926 14:50:04.980655 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:04.980691 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:04.981505 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:04.981703 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:04.981706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:04.981928 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:04.982093 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:04.982097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.982729 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.982730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:04.983535 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:04.983537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:04.983853 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:04.983853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:04.983946 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:04.984465 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:04.984476 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:04.986203 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:04.987782 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:04.987788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:04.987790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:04.988014 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:04.988220 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:04.988221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:04.988221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:04.988318 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:04.988564 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:04.988569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:04.988693 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 14:50:04.988795 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:50:04.988932 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260926 14:50:04.989558 6211 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38249
I20260926 14:50:04.989638 6211 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:04.989753 6211 heartbeater.cc:511] Master 127.5.15.190:38249 requested a full tablet report, sending...
I20260926 14:50:04.990041 5752 ts_manager.cc:194] Registered new tserver with Master: 676f91ba096246a1b2d14679e355e510 (127.5.15.131:36111)
I20260926 14:50:04.990376 5752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:39897
I20260926 14:50:04.997844 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:05.004797 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:04.999456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.5.15.190:38249)
0926 14:50:04.999537 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:04.999541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:04.999606 (+ 65us) server_negotiation.cc:407] Connection header received
0926 14:50:04.999640 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:04.999641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:04.999648 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:04.999693 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:05.000183 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.000603 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.001294 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.001460 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.001508 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.002538 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:05.002539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:05.002539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:05.002544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:05.003849 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.004127 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.004128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.004128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.004173 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.004306 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.004306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.004307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.004415 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:05.004498 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.004630 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.004715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:50:05.006867 5752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:05.008345 5752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:05.014302 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.010289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:44203 (local address 127.0.0.1:39648)
0926 14:50:05.010638 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:05.010648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:05.010685 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:05.011036 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:05.011037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:05.011040 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:05.011085 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.011088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.012035 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.012036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.013276 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.013278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.013999 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.013999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.014074 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.014086 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 14:50:05.014181 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 14:50:05.014232 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":25,"thread_start_us":65,"threads_started":1}
I20260926 14:50:05.014307 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.010680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.5.15.130:44203)
0926 14:50:05.010859 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.010863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.010873 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:05.010916 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.010918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.010931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.011032 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:05.011150 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.011928 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.013370 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.013915 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.013999 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.014055 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.014198 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.014235 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260926 14:50:05.014446 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.010294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41477 (local address 127.0.0.1:44470)
0926 14:50:05.010797 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:05.010805 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:05.010848 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:05.011167 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:05.011167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:05.011169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:05.011209 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.011210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.012382 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.012382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.013208 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.013210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.014179 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.014180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.014248 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.014252 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:05.014294 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 14:50:05.014338 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":66,"threads_started":1}
I20260926 14:50:05.014446 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.010289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:36111 (local address 127.0.0.1:59560)
0926 14:50:05.010453 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:05.010489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:05.010553 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:05.010972 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:05.010975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:05.010977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:05.011042 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.011044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.012317 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.012318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.013417 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.013418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.014245 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.014246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.014304 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.014306 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:05.014333 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 14:50:05.014362 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":25,"thread_start_us":66,"threads_started":1}
I20260926 14:50:05.014688 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.010509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.5.15.131:36111)
0926 14:50:05.010665 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.010669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.010678 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:05.010715 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.010717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.010728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.010930 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:05.011128 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.012216 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.013493 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.014161 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.014485 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.014549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.014575 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.014618 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260926 14:50:05.014767 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.010464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.5.15.129:41477)
0926 14:50:05.010622 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.010626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.010909 (+ 283us) server_negotiation.cc:407] Connection header received
0926 14:50:05.010980 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.010982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.010996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.011067 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:05.011282 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.012302 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.013295 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.014022 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.014091 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.014143 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.014628 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.014667 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":99,"threads_started":1}
I20260926 14:50:05.015322 5877 tablet_service.cc:1467] Processing CreateTablet for tablet cb833350b12e487691d146843178fd86 (DEFAULT_TABLE table=test-table [id=e3ba583b0b4d44329cecc5183e40b258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:05.015372 6145 tablet_service.cc:1467] Processing CreateTablet for tablet cb833350b12e487691d146843178fd86 (DEFAULT_TABLE table=test-table [id=e3ba583b0b4d44329cecc5183e40b258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:05.015700 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb833350b12e487691d146843178fd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:05.015702 6145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb833350b12e487691d146843178fd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:05.018373 6225 tablet_bootstrap.cc:492] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: Bootstrap starting.
I20260926 14:50:05.018714 6011 tablet_service.cc:1467] Processing CreateTablet for tablet cb833350b12e487691d146843178fd86 (DEFAULT_TABLE table=test-table [id=e3ba583b0b4d44329cecc5183e40b258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:05.019073 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb833350b12e487691d146843178fd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:05.019100 6225 tablet_bootstrap.cc:654] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:05.019629 6225 log.cc:824] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:05.020490 6225 tablet_bootstrap.cc:492] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: No bootstrap required, opened a new log
I20260926 14:50:05.020687 6225 ts_tablet_manager.cc:1397] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:05.022446 6228 tablet_bootstrap.cc:492] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: Bootstrap starting.
I20260926 14:50:05.022537 6226 tablet_bootstrap.cc:492] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: Bootstrap starting.
I20260926 14:50:05.022795 6225 raft_consensus.cc:348] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.022990 6225 raft_consensus.cc:374] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:05.023037 6225 raft_consensus.cc:729] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d876da6de214059a832e83f825b7c63, State: Initialized, Role: FOLLOWER
I20260926 14:50:05.023147 6228 tablet_bootstrap.cc:654] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:05.023151 6226 tablet_bootstrap.cc:654] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:05.023177 6225 consensus_queue.cc:260] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.023479 6225 ts_tablet_manager.cc:1428] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:05.023494 5943 heartbeater.cc:503] Master 127.5.15.190:38249 was elected leader, sending a full tablet report...
I20260926 14:50:05.023725 6226 log.cc:824] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:05.023949 6228 log.cc:824] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:05.024597 6226 tablet_bootstrap.cc:492] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: No bootstrap required, opened a new log
I20260926 14:50:05.024667 6226 ts_tablet_manager.cc:1397] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:05.024801 6228 tablet_bootstrap.cc:492] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: No bootstrap required, opened a new log
I20260926 14:50:05.024901 6228 ts_tablet_manager.cc:1397] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:05.026036 6226 raft_consensus.cc:348] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.026180 6226 raft_consensus.cc:374] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:05.026223 6226 raft_consensus.cc:729] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676f91ba096246a1b2d14679e355e510, State: Initialized, Role: FOLLOWER
I20260926 14:50:05.026172 6228 raft_consensus.cc:348] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.026350 6228 raft_consensus.cc:374] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:05.026330 6226 consensus_queue.cc:260] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.026391 6228 raft_consensus.cc:729] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727a6ab0b10f404bbeedaa0b1eeef68e, State: Initialized, Role: FOLLOWER
I20260926 14:50:05.026496 6228 consensus_queue.cc:260] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.026611 6226 ts_tablet_manager.cc:1428] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:05.026616 6211 heartbeater.cc:503] Master 127.5.15.190:38249 was elected leader, sending a full tablet report...
I20260926 14:50:05.026798 6228 ts_tablet_manager.cc:1428] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:05.026820 6077 heartbeater.cc:503] Master 127.5.15.190:38249 was elected leader, sending a full tablet report...
W20260926 14:50:05.070814 6078 tablet.cc:2369] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:05.174544 5944 tablet.cc:2369] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:05.230532 6212 tablet.cc:2369] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:05.355842 6232 raft_consensus.cc:482] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:05.355962 6232 raft_consensus.cc:504] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.356390 6232 leader_election.cc:288] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 727a6ab0b10f404bbeedaa0b1eeef68e (127.5.15.130:44203), 2d876da6de214059a832e83f825b7c63 (127.5.15.129:41477)
I20260926 14:50:05.360242 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.356714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:49007 (local address 127.5.15.129:41477)
0926 14:50:05.356756 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.356758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.356840 (+ 82us) server_negotiation.cc:407] Connection header received
0926 14:50:05.356935 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.356936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.356946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.356998 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:05.357114 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.357797 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.359325 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.359880 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.359960 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.359969 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.360140 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.360176 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:05.360242 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.356661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:41487 (local address 127.5.15.130:44203)
0926 14:50:05.356723 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.356726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.356737 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:05.356797 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.356798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.356809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.356894 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:05.357056 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.357798 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.359324 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.359880 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.359961 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.359969 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.360140 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.360174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:05.360275 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.356605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:44203 (local address 127.5.15.131:41487)
0926 14:50:05.356677 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:05.356688 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:05.356735 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:05.356910 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:05.356910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:05.356912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:05.356984 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.356986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.357878 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.357879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.359235 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.359237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.359968 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.359969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.360049 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.360051 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:05.360108 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:50:05.360167 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":65}
I20260926 14:50:05.360280 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.356605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41477 (local address 127.5.15.131:49007)
0926 14:50:05.356780 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:05.356788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:05.356843 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:05.357002 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:05.357002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:05.357004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:05.357060 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.357061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.357886 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.357887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.359235 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:05.359237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.359968 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.359969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:05.360047 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.360051 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:05.360108 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:50:05.360167 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":87,"thread_start_us":105,"threads_started":1}
I20260926 14:50:05.360647 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb833350b12e487691d146843178fd86" candidate_uuid: "676f91ba096246a1b2d14679e355e510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" is_pre_election: true
I20260926 14:50:05.360648 5897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb833350b12e487691d146843178fd86" candidate_uuid: "676f91ba096246a1b2d14679e355e510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d876da6de214059a832e83f825b7c63" is_pre_election: true
I20260926 14:50:05.360816 6031 raft_consensus.cc:2454] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676f91ba096246a1b2d14679e355e510 in term 0.
I20260926 14:50:05.360816 5897 raft_consensus.cc:2454] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676f91ba096246a1b2d14679e355e510 in term 0.
I20260926 14:50:05.361074 6101 leader_election.cc:302] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 676f91ba096246a1b2d14679e355e510, 727a6ab0b10f404bbeedaa0b1eeef68e; no voters:
I20260926 14:50:05.361285 6232 raft_consensus.cc:2781] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:05.361343 6232 raft_consensus.cc:482] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:05.361358 6232 raft_consensus.cc:3037] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:05.361994 6232 raft_consensus.cc:504] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.362135 6232 leader_election.cc:288] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [CANDIDATE]: Term 1 election: Requested vote from peers 727a6ab0b10f404bbeedaa0b1eeef68e (127.5.15.130:44203), 2d876da6de214059a832e83f825b7c63 (127.5.15.129:41477)
I20260926 14:50:05.362275 6031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb833350b12e487691d146843178fd86" candidate_uuid: "676f91ba096246a1b2d14679e355e510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e"
I20260926 14:50:05.362284 5897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb833350b12e487691d146843178fd86" candidate_uuid: "676f91ba096246a1b2d14679e355e510" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d876da6de214059a832e83f825b7c63"
I20260926 14:50:05.362361 6031 raft_consensus.cc:3037] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:05.362362 5897 raft_consensus.cc:3037] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:05.363265 5897 raft_consensus.cc:2454] T cb833350b12e487691d146843178fd86 P 2d876da6de214059a832e83f825b7c63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676f91ba096246a1b2d14679e355e510 in term 1.
I20260926 14:50:05.363284 6031 raft_consensus.cc:2454] T cb833350b12e487691d146843178fd86 P 727a6ab0b10f404bbeedaa0b1eeef68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676f91ba096246a1b2d14679e355e510 in term 1.
I20260926 14:50:05.363435 6099 leader_election.cc:302] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d876da6de214059a832e83f825b7c63, 676f91ba096246a1b2d14679e355e510; no voters:
I20260926 14:50:05.363626 6232 raft_consensus.cc:2781] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:05.363801 6232 raft_consensus.cc:686] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [term 1 LEADER]: Becoming Leader. State: Replica: 676f91ba096246a1b2d14679e355e510, State: Running, Role: LEADER
I20260926 14:50:05.363900 6232 consensus_queue.cc:237] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } }
I20260926 14:50:05.364697 5752 catalog_manager.cc:5495] T cb833350b12e487691d146843178fd86 P 676f91ba096246a1b2d14679e355e510 reported cstate change: term changed from 0 to 1, leader changed from <none> to 676f91ba096246a1b2d14679e355e510 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "676f91ba096246a1b2d14679e355e510" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727a6ab0b10f404bbeedaa0b1eeef68e" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d876da6de214059a832e83f825b7c63" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "676f91ba096246a1b2d14679e355e510" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 36111 } health_report { overall_health: HEALTHY } } }
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:05.459592 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.451811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.5.15.129:41477)
0926 14:50:05.451876 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.451878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.452475 (+ 597us) server_negotiation.cc:407] Connection header received
0926 14:50:05.452505 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.452506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.452519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.452586 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:05.453268 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.453751 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.454461 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.454653 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.454736 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.456940 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:05.456950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:05.456951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:05.456957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:05.458584 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.458982 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.458985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.458986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.459046 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.459192 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.459192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.459192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.459311 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:05.459412 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.459465 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.459526 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:05.481889 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.475247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.5.15.129:41477)
0926 14:50:05.475299 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.475301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.475421 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:50:05.475468 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.475468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.475478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.475530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:05.476037 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.476514 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.477192 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.477367 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.477415 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.479566 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:05.479567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:05.479567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:05.479573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:05.480731 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.481238 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.481239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.481239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.481293 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.481474 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.481474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.481474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.481560 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:05.481632 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.481756 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.481818 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[5707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[5707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:05.502692 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:05.496476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.5.15.129:41477)
0926 14:50:05.496562 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:05.496564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:05.496642 (+ 78us) server_negotiation.cc:407] Connection header received
0926 14:50:05.496667 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:05.496668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:05.496676 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:05.496729 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:05.497200 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.497651 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.498358 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:05.498521 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:05.498570 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:05.500622 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:05.500623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:05.500624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:05.500629 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:05.501744 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.502124 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.502126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.502126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.502183 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:05.502334 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:05.502334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:05.502335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:05.502420 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:05.502482 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:05.502564 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:50:05.502612 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 14:50:05.503142 5857 tablet_service.cc:2908] Rejecting scan request for tablet cb833350b12e487691d146843178fd86: Uninitialized: clean time has not yet been initialized
I20260926 14:50:05.503944 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5814
I20260926 14:50:05.508719 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5948
I20260926 14:50:05.513408 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6082
I20260926 14:50:05.517550 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 5720
2026-09-26T14:50:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2307 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[6249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[6249](info): set up 2 sockets
Sep 26 14:50:05 dist-test-slave-s4p1 krb5kdc[6249](info): commencing operation
krb5kdc: starting...
W20260926 14:50:06.538301 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:06 dist-test-slave-s4p1 krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:06.648890 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40527
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:06.726737 6262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:06.726987 6262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:06.727027 6262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:06.728133 6262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:06.728199 6262 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:50:06.728219 6262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:06.728241 6262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:06.728273 6262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:06.728297 6262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:06.729694 6262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:40527
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:06.729969 6262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:06.730183 6262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:06.734798 6267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.734846 6262 server_base.cc:1034] running on GCE node
W20260926 14:50:06.734815 6268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:06.734800 6270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.735291 6262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:06.735600 6262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:06.736737 6262 hybrid_clock.cc:648] HybridClock initialized: now 1790434206736728 us; error 32 us; skew 500 ppm
Sep 26 14:50:06 dist-test-slave-s4p1 krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:06.744692 6262 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:06.745147 6262 webserver.cc:460] Webserver started at http://127.5.15.190:42133/ using document root <none> and password file <none>
I20260926 14:50:06.745368 6262 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:06.745412 6262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:06.745563 6262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:06.746572 6262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "814141e15bc144fe8fd96a1c519de046"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-s4p1"
server_key: "1430317e9dd959945b5a7ac1bc93b2df"
server_key_iv: "ad2170c5d4c66b5ffda47f873449102e"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.746949 6262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "814141e15bc144fe8fd96a1c519de046"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-s4p1"
server_key: "1430317e9dd959945b5a7ac1bc93b2df"
server_key_iv: "ad2170c5d4c66b5ffda47f873449102e"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.748193 6262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:06.748883 6277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.749102 6262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:06.749245 6262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "814141e15bc144fe8fd96a1c519de046"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-s4p1"
server_key: "1430317e9dd959945b5a7ac1bc93b2df"
server_key_iv: "ad2170c5d4c66b5ffda47f873449102e"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.749316 6262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:06.760753 6262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:06.762425 6262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:06.762579 6262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:06.767119 6262 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:40527
I20260926 14:50:06.767164 6339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:40527 every 8 connection(s)
I20260926 14:50:06.767467 6262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:06.768026 6340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:06.770160 6340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: Bootstrap starting.
I20260926 14:50:06.770711 6340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:06.771112 6340 log.cc:824] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:06.771770 6340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: No bootstrap required, opened a new log
I20260926 14:50:06.772939 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6262
I20260926 14:50:06.773016 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:06.773195 5182 external_mini_cluster.cc:1428] Setting key 3e1a1b54b7f373be717050eb96b998f5
I20260926 14:50:06.774284 6340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814141e15bc144fe8fd96a1c519de046" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40527 } }
I20260926 14:50:06.774451 6340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:06.774487 6340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 814141e15bc144fe8fd96a1c519de046, State: Initialized, Role: FOLLOWER
I20260926 14:50:06.774653 6340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814141e15bc144fe8fd96a1c519de046" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40527 } }
I20260926 14:50:06.774740 6340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:06.774801 6340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:06.774859 6340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:06.775431 6340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814141e15bc144fe8fd96a1c519de046" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40527 } }
I20260926 14:50:06.775532 6340 leader_election.cc:302] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 814141e15bc144fe8fd96a1c519de046; no voters:
I20260926 14:50:06.775673 6340 leader_election.cc:288] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:06.775727 6345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:06.775904 6340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:06.775918 6345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [term 1 LEADER]: Becoming Leader. State: Replica: 814141e15bc144fe8fd96a1c519de046, State: Running, Role: LEADER
I20260926 14:50:06.776141 6345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814141e15bc144fe8fd96a1c519de046" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40527 } }
I20260926 14:50:06.776467 6345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 814141e15bc144fe8fd96a1c519de046. Latest consensus state: current_term: 1 leader_uuid: "814141e15bc144fe8fd96a1c519de046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814141e15bc144fe8fd96a1c519de046" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40527 } } }
I20260926 14:50:06.776481 6347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "814141e15bc144fe8fd96a1c519de046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "814141e15bc144fe8fd96a1c519de046" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40527 } } }
I20260926 14:50:06.776582 6345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:06.776664 6347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:06.776872 6354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:06.777391 6354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:06.778875 6354 catalog_manager.cc:1347] Generated new cluster ID: 0d3a0c19818248228edb2a305d2eeb2f
I20260926 14:50:06.778928 6354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:06 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:06.784937 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.773735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.5.15.190:40527)
0926 14:50:06.773960 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.773971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.774007 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:50:06.774250 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.774258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.774317 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.774471 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.775266 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.776842 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.777965 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.778495 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.778611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.781414 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.781418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.781418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.781429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.783374 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.783793 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.783795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.783796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.783910 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.784120 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.784121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.784121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.784387 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.784530 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.784634 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.784755 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260926 14:50:06.787284 6354 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:06.787711 6354 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:06.791220 6354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: Generated new TSK 0
I20260926 14:50:06.791378 6354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:06.821393 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:40527
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:06.922648 6366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:06.922943 6366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:06.922986 6366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:06.924146 6366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:06.924211 6366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:06.924247 6366 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:06.924283 6366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:06.925683 6366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:06.925949 6366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:06.926179 6366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:06.931779 6371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:06.931820 6374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:06.931785 6372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:06.932094 6366 server_base.cc:1034] running on GCE node
I20260926 14:50:06.932256 6366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:06.932507 6366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:06.933662 6366 hybrid_clock.cc:648] HybridClock initialized: now 1790434206933645 us; error 34 us; skew 500 ppm
Sep 26 14:50:06 dist-test-slave-s4p1 krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:06.936162 6366 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:06.936574 6366 webserver.cc:460] Webserver started at http://127.5.15.129:34547/ using document root <none> and password file <none>
I20260926 14:50:06.936774 6366 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:06.936820 6366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:06.936900 6366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:06.937852 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "9ed7062da196487fb4b98d621ed5cdae"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-s4p1"
server_key: "d5a4f98fa81517661bfc5bbcb55c4c31"
server_key_iv: "396dc1dbe20eb6b9d311683c0e8d4f2a"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.938136 6366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "9ed7062da196487fb4b98d621ed5cdae"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-s4p1"
server_key: "d5a4f98fa81517661bfc5bbcb55c4c31"
server_key_iv: "396dc1dbe20eb6b9d311683c0e8d4f2a"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.939463 6366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:06.940137 6381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.940321 6366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.940390 6366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "9ed7062da196487fb4b98d621ed5cdae"
format_stamp: "Formatted at 2026-09-26 14:50:06 on dist-test-slave-s4p1"
server_key: "d5a4f98fa81517661bfc5bbcb55c4c31"
server_key_iv: "396dc1dbe20eb6b9d311683c0e8d4f2a"
server_key_version: "encryptionkey@0"
I20260926 14:50:06.940476 6366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:06.956933 6366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:06.959213 6366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:06.959482 6366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:06.960074 6366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:06.960124 6366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:06.960144 6366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:06.960155 6366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:06 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:06.967289 6366 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:35053
I20260926 14:50:06.967677 6366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:06.968091 6496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:35053 every 8 connection(s)
I20260926 14:50:06.969664 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.960486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:43041 (local address 127.5.15.190:40527)
0926 14:50:06.960549 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.960551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.960748 (+ 197us) server_negotiation.cc:407] Connection header received
0926 14:50:06.961282 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.961283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.961294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.961371 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.962033 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.962886 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.963994 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.964222 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.964308 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.966869 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.966870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.966871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.966878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.968535 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.968886 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.968886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.968887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.968937 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.969160 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.969160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.969161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.969272 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.969341 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.969543 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.969605 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:06.969882 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.960465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.5.15.129:43041)
0926 14:50:06.960633 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:06.960666 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:06.961146 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:06.961448 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:06.961452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:06.961746 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:06.961938 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.961943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.962978 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.962979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.963899 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.963901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.964317 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.964318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.964436 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.964948 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:06.964956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:06.966699 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:06.968606 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.968610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.968611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.968811 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.969007 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.969008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.969008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.969096 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.969347 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:06.969351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:06.969452 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 14:50:06.969530 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:50:06.969638 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260926 14:50:06.970360 6497 heartbeater.cc:347] Connected to a master server at 127.5.15.190:40527
I20260926 14:50:06.970458 6497 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:06.970644 6497 heartbeater.cc:511] Master 127.5.15.190:40527 requested a full tablet report, sending...
I20260926 14:50:06.971262 6293 ts_manager.cc:194] Registered new tserver with Master: 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053)
I20260926 14:50:06.971909 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:43041
I20260926 14:50:06.976557 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.971377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.5.15.129:51625)
0926 14:50:06.971486 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:06.971496 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:06.971546 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:06.971659 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:06.971660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:06.971849 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:06.971900 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.971901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.972769 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.972770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.973295 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:06.973296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.973639 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.973640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:06.973716 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.974028 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:06.974032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:06.974282 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:06.975613 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.975614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.975614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.975797 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.976014 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:06.976015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:06.976015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:06.976064 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:06.976314 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:06.976314 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:06.976358 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 14:50:06.976426 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:50:06.976495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260926 14:50:06.976620 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6366
I20260926 14:50:06.976682 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:06.971521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:51625 (local address 127.5.15.190:40527)
0926 14:50:06.971556 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:06.971558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:06.971567 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:06.971588 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:06.971589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:06.971601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:06.971655 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:06.971989 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.972680 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.973386 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:06.973573 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:06.973641 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:06.974384 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:06.974386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:06.974386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:06.974393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:06.975528 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.975881 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.975886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.975887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.975936 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:06.976115 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:06.976116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:06.976116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:06.976254 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:06.976312 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:06.976509 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 14:50:06.976587 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:06.976703 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:06.976912 5182 external_mini_cluster.cc:1428] Setting key ff8ed3a5823f3d4c31d671969f76661b
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:07.000010 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:40527
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:07.080528 6506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:07.080717 6506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:07.080741 6506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:07.081818 6506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:07.081849 6506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:07.081868 6506 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:07.081885 6506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:07.083176 6506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:07.083362 6506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:07.083520 6506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:07.088361 6512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.088356 6514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.088352 6511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:07.088467 6506 server_base.cc:1034] running on GCE node
I20260926 14:50:07.088827 6506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:07.089080 6506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:07.090236 6506 hybrid_clock.cc:648] HybridClock initialized: now 1790434207090219 us; error 61 us; skew 500 ppm
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:07.092670 6506 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:07.093081 6506 webserver.cc:460] Webserver started at http://127.5.15.130:44715/ using document root <none> and password file <none>
I20260926 14:50:07.093299 6506 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:07.093365 6506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:07.093509 6506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:07.094512 6506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "49e02f50502d4025a6ff91ef602209c7"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-s4p1"
server_key: "65b3d337cac84d112136ed676fc6c11f"
server_key_iv: "a8360d72c066e51c1d130adf38f45fcb"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.094916 6506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "49e02f50502d4025a6ff91ef602209c7"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-s4p1"
server_key: "65b3d337cac84d112136ed676fc6c11f"
server_key_iv: "a8360d72c066e51c1d130adf38f45fcb"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.096177 6506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:07.096812 6521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.096978 6506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.097069 6506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "49e02f50502d4025a6ff91ef602209c7"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-s4p1"
server_key: "65b3d337cac84d112136ed676fc6c11f"
server_key_iv: "a8360d72c066e51c1d130adf38f45fcb"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.097144 6506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:07.108726 6506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:07.110325 6506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:07.110462 6506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:07.111060 6506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:07.111109 6506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.111137 6506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:07.111158 6506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:07.117972 6506 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:43235
I20260926 14:50:07.118040 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:43235 every 8 connection(s)
I20260926 14:50:07.118341 6506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:07.120687 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.111408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:47351 (local address 127.5.15.190:40527)
0926 14:50:07.111452 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.111455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.111656 (+ 201us) server_negotiation.cc:407] Connection header received
0926 14:50:07.112172 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.112173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.112188 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.112259 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.112989 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.113789 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.115238 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.115519 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.115602 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.118145 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.118147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.118147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.118155 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.119636 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.120004 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.120005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.120005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.120051 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.120242 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.120243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.120243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.120338 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.120384 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.120572 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.120631 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:07.120899 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.111310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.5.15.130:47351)
0926 14:50:07.111519 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.111557 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.112057 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.112356 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.112359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.112648 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:07.112906 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.112912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.113881 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.113882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.115142 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.115144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.115626 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.115628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.115764 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.116218 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:07.116232 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:07.118052 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:07.119704 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.119708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.119710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.119929 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.120108 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.120108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.120108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.120193 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.120389 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:07.120392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:07.120486 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.120559 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.120658 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260926 14:50:07.121361 6637 heartbeater.cc:347] Connected to a master server at 127.5.15.190:40527
I20260926 14:50:07.121436 6637 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:07.121595 6637 heartbeater.cc:511] Master 127.5.15.190:40527 requested a full tablet report, sending...
I20260926 14:50:07.121963 6293 ts_manager.cc:194] Registered new tserver with Master: 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.122475 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:47351
I20260926 14:50:07.124543 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6506
I20260926 14:50:07.124655 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:07.124845 5182 external_mini_cluster.cc:1428] Setting key 4f99f91de0e2673b0b1cc74d45eceb35
I20260926 14:50:07.127861 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.122207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.5.15.130:46305)
0926 14:50:07.122321 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.122330 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.122378 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.122516 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.122516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.122698 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:07.122765 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.122766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.123503 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.123503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.123940 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.123941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.124202 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.124202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.124249 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.124620 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:07.124626 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:07.125019 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:07.126809 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.126810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.126810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.127046 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.127229 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.127230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.127231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.127296 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.127532 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:07.127533 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:07.127574 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.127667 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.127758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260926 14:50:07.127856 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.122283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:46305 (local address 127.5.15.190:40527)
0926 14:50:07.122317 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.122319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.122372 (+ 53us) server_negotiation.cc:407] Connection header received
0926 14:50:07.122437 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.122438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.122444 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.122481 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.122863 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.123428 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.124000 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.124148 (+ 148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.124190 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.125157 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.125158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.125158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.125164 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.126701 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.127120 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.127121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.127121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.127167 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.127351 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.127351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.127351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.127461 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.127535 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.127690 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.127781 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:07.149034 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:40527
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:39617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:07.228125 6646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:07.228305 6646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:07.228327 6646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:07.229422 6646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:07.229462 6646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:07.229481 6646 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:07.229553 6646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:07.230916 6646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:39617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:40527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:07.231125 6646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:07.231344 6646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:07.236040 6654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.236092 6651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:07.236047 6652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:07.236276 6646 server_base.cc:1034] running on GCE node
I20260926 14:50:07.236446 6646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:07.236634 6646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:07.237870 6646 hybrid_clock.cc:648] HybridClock initialized: now 1790434207237858 us; error 25 us; skew 500 ppm
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:07.240275 6646 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:07.240617 6646 webserver.cc:460] Webserver started at http://127.5.15.131:40923/ using document root <none> and password file <none>
I20260926 14:50:07.240772 6646 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:07.240809 6646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:07.240906 6646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:07.241803 6646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "e09c9092e0c7449085f3e8316c101294"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-s4p1"
server_key: "f1f973fe7071203f8db9cfab6c5723cd"
server_key_iv: "e8e1d6a29918dc84018e197a44b66849"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.242087 6646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "e09c9092e0c7449085f3e8316c101294"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-s4p1"
server_key: "f1f973fe7071203f8db9cfab6c5723cd"
server_key_iv: "e8e1d6a29918dc84018e197a44b66849"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.243417 6646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:07.244091 6661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.244310 6646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:07.244385 6646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "e09c9092e0c7449085f3e8316c101294"
format_stamp: "Formatted at 2026-09-26 14:50:07 on dist-test-slave-s4p1"
server_key: "f1f973fe7071203f8db9cfab6c5723cd"
server_key_iv: "e8e1d6a29918dc84018e197a44b66849"
server_key_version: "encryptionkey@0"
I20260926 14:50:07.244432 6646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:07.259346 6646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:07.260926 6646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:07.261042 6646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:07.261462 6646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:07.261524 6646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.261559 6646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:07.261608 6646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.268206 6646 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:37769
I20260926 14:50:07.268265 6776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:37769 every 8 connection(s)
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:07.268572 6646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:07.271137 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.261857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:39941 (local address 127.5.15.190:40527)
0926 14:50:07.261924 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.261927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.262077 (+ 150us) server_negotiation.cc:407] Connection header received
0926 14:50:07.262507 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.262508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.262521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.262605 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.263428 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.264273 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.265802 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.266139 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.266214 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.268794 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.268796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.268797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.268805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.269951 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.270337 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.270337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.270338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.270385 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.270580 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.270581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.270581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.270694 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.270738 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.271017 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.271078 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:07.271252 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.261823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.5.15.131:39941)
0926 14:50:07.261955 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.261993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.262419 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.262648 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.262651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.262997 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:07.263324 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.263331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.264379 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.264380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.265694 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.265697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.266249 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.266250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.266386 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.266898 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:07.266906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:07.268689 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:07.270012 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.270017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.270019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.270273 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.270426 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.270427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.270427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.270511 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.270770 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:07.270775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:07.270905 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.270983 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.271086 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260926 14:50:07.271710 6777 heartbeater.cc:347] Connected to a master server at 127.5.15.190:40527
I20260926 14:50:07.271886 6777 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:07.272083 6777 heartbeater.cc:511] Master 127.5.15.190:40527 requested a full tablet report, sending...
I20260926 14:50:07.272457 6293 ts_manager.cc:194] Registered new tserver with Master: e09c9092e0c7449085f3e8316c101294 (127.5.15.131:37769)
I20260926 14:50:07.272948 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:39941
I20260926 14:50:07.273844 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6646
I20260926 14:50:07.273943 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:07.274111 5182 external_mini_cluster.cc:1428] Setting key dbd359d45a5b0a15a793e581467d09e7
I20260926 14:50:07.274976 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:07.278508 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.272822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:51083 (local address 127.5.15.190:40527)
0926 14:50:07.272865 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.272867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.272925 (+ 58us) server_negotiation.cc:407] Connection header received
0926 14:50:07.272993 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.272994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.273004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.273049 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.273367 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.273856 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.274837 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.275038 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.275111 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.276027 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.276028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.276028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.276035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.277411 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.277785 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.277786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.277786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.277842 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.278034 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.278035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.278036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.278132 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.278204 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.278356 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.278425 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:50:07.278522 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.272756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.5.15.131:51083)
0926 14:50:07.272873 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.272884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.272939 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.273071 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.273072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.273233 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:07.273305 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.273306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.274001 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.274002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.274712 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.274714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.275117 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.275118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.275191 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.275614 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:07.275618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:07.275947 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:07.277513 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.277514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.277515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.277721 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.277907 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.277908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.277908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.277964 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.278196 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:07.278197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:07.278239 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.278333 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.278426 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:07.296976 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.290719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.5.15.190:40527)
0926 14:50:07.290780 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.290781 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.290808 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:50:07.290890 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.290891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.290898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.290949 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.291444 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.291982 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.292643 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.292808 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.292864 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.294809 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.294810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.294810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.294815 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.296138 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.296407 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.296408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.296408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.296452 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.296587 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.296587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.296588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.296661 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.296725 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.296797 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.296854 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:07.298312 6293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:07.300088 6293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:07.306130 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.302190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37769 (local address 127.0.0.1:43120)
0926 14:50:07.302505 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.302514 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.302555 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.302795 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.302796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.302798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.302850 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.302852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.304080 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.304080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.305179 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.305181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.305841 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.305917 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.305929 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.305996 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.306043 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":30,"thread_start_us":82,"threads_started":1}
I20260926 14:50:07.306241 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.302427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.5.15.130:43235)
0926 14:50:07.302812 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.302817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.302826 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:07.302862 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.302864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.302878 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.302945 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.303126 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.303810 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305221 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.305921 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305987 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.306039 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.306111 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.306148 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":70,"threads_started":1}
I20260926 14:50:07.306357 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.302280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.5.15.129:35053)
0926 14:50:07.302449 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.302453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.302463 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:07.302565 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.302567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.302601 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.302672 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.302873 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.304006 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.304985 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.305736 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305783 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.305836 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.306061 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.306100 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260926 14:50:07.306399 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.302201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:43235 (local address 127.0.0.1:58076)
0926 14:50:07.302578 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.302608 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.302658 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.303011 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.303012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.303014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.303051 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.303052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.303895 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.303896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.305134 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.305136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.306000 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.306000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.306048 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.306052 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.306102 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.306130 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":69,"threads_started":1}
I20260926 14:50:07.306439 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.302402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.5.15.131:37769)
0926 14:50:07.302561 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.302566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.302576 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:07.302632 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.302634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.302649 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.302721 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.302947 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.303990 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305258 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.305736 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305837 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.305891 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.306307 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.306346 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260926 14:50:07.306586 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.302190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:35053 (local address 127.0.0.1:46938)
0926 14:50:07.302347 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.302384 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.302447 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.302672 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.302675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.302677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.302732 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.302735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.304090 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.304091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.304892 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.304895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.305897 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.305897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.305971 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.305976 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.306462 (+ 486us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.306516 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":58,"threads_started":1}
I20260926 14:50:07.307030 6571 tablet_service.cc:1467] Processing CreateTablet for tablet eb3940b1b4bd45038379d4284ed8c0f7 (DEFAULT_TABLE table=test-table [id=0d784a840cf1446badd2020487aafda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:07.307046 6711 tablet_service.cc:1467] Processing CreateTablet for tablet eb3940b1b4bd45038379d4284ed8c0f7 (DEFAULT_TABLE table=test-table [id=0d784a840cf1446badd2020487aafda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:07.307277 6571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3940b1b4bd45038379d4284ed8c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.307358 6711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3940b1b4bd45038379d4284ed8c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.309576 6800 tablet_bootstrap.cc:492] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Bootstrap starting.
I20260926 14:50:07.309908 6801 tablet_bootstrap.cc:492] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Bootstrap starting.
I20260926 14:50:07.310212 6431 tablet_service.cc:1467] Processing CreateTablet for tablet eb3940b1b4bd45038379d4284ed8c0f7 (DEFAULT_TABLE table=test-table [id=0d784a840cf1446badd2020487aafda2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:07.310495 6801 tablet_bootstrap.cc:654] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.310529 6431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb3940b1b4bd45038379d4284ed8c0f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.310608 6800 tablet_bootstrap.cc:654] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.310837 6801 log.cc:824] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:07.310961 6800 log.cc:824] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:07.311573 6801 tablet_bootstrap.cc:492] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: No bootstrap required, opened a new log
I20260926 14:50:07.311654 6801 ts_tablet_manager.cc:1397] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:07.311779 6800 tablet_bootstrap.cc:492] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: No bootstrap required, opened a new log
I20260926 14:50:07.311868 6800 ts_tablet_manager.cc:1397] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:07.312984 6801 raft_consensus.cc:348] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.313130 6801 raft_consensus.cc:374] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.313194 6801 raft_consensus.cc:729] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e02f50502d4025a6ff91ef602209c7, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.313326 6801 consensus_queue.cc:260] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.313614 6637 heartbeater.cc:503] Master 127.5.15.190:40527 was elected leader, sending a full tablet report...
I20260926 14:50:07.313894 6805 tablet_bootstrap.cc:492] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Bootstrap starting.
I20260926 14:50:07.313886 6800 raft_consensus.cc:348] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.314035 6800 raft_consensus.cc:374] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.314098 6800 raft_consensus.cc:729] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09c9092e0c7449085f3e8316c101294, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.314188 6800 consensus_queue.cc:260] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.313601 6801 ts_tablet_manager.cc:1428] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:07.314429 6800 ts_tablet_manager.cc:1428] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 14:50:07.314433 6777 heartbeater.cc:503] Master 127.5.15.190:40527 was elected leader, sending a full tablet report...
I20260926 14:50:07.314939 6805 tablet_bootstrap.cc:654] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.315269 6805 log.cc:824] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:07.315893 6805 tablet_bootstrap.cc:492] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: No bootstrap required, opened a new log
I20260926 14:50:07.315944 6805 ts_tablet_manager.cc:1397] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:07.317214 6805 raft_consensus.cc:348] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.317327 6805 raft_consensus.cc:374] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.317381 6805 raft_consensus.cc:729] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed7062da196487fb4b98d621ed5cdae, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.317489 6805 consensus_queue.cc:260] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.317730 6805 ts_tablet_manager.cc:1428] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:07.317763 6497 heartbeater.cc:503] Master 127.5.15.190:40527 was elected leader, sending a full tablet report...
I20260926 14:50:07.364023 6808 raft_consensus.cc:482] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:07.364163 6808 raft_consensus.cc:504] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.364559 6808 leader_election.cc:288] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e09c9092e0c7449085f3e8316c101294 (127.5.15.131:37769), 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.367484 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.364759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:43235 (local address 127.5.15.129:40969)
0926 14:50:07.364804 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.364815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.364869 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.365012 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.365013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.365016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.365087 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.365088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.365836 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.365836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.366614 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.366616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.367205 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.367206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.367294 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.367299 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.367351 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.367401 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 14:50:07.367494 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.364783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:40969 (local address 127.5.15.130:43235)
0926 14:50:07.364850 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.364851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.364862 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:07.364917 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.364917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.364926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.364990 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.365181 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.365743 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.366697 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.367135 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.367191 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.367198 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.367367 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.367421 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:07.367585 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.364780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:40345 (local address 127.5.15.131:37769)
0926 14:50:07.364821 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.364823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.365141 (+ 318us) server_negotiation.cc:407] Connection header received
0926 14:50:07.365173 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.365174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.365185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.365235 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.365386 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.365874 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.366821 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.367215 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.367288 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.367295 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.367473 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.367516 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:07.367781 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.364763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37769 (local address 127.5.15.129:40345)
0926 14:50:07.365061 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.365070 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.365115 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.365254 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.365255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.365257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.365323 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.365325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.365956 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.365956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.366698 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.366700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.367293 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.367294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.367399 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.367403 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.367457 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.367492 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":151,"thread_start_us":97,"threads_started":1}
I20260926 14:50:07.368111 6731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3940b1b4bd45038379d4284ed8c0f7" candidate_uuid: "9ed7062da196487fb4b98d621ed5cdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09c9092e0c7449085f3e8316c101294" is_pre_election: true
I20260926 14:50:07.368110 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3940b1b4bd45038379d4284ed8c0f7" candidate_uuid: "9ed7062da196487fb4b98d621ed5cdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e02f50502d4025a6ff91ef602209c7" is_pre_election: true
I20260926 14:50:07.368257 6731 raft_consensus.cc:2454] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed7062da196487fb4b98d621ed5cdae in term 0.
I20260926 14:50:07.368274 6591 raft_consensus.cc:2454] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed7062da196487fb4b98d621ed5cdae in term 0.
I20260926 14:50:07.368467 6383 leader_election.cc:302] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e02f50502d4025a6ff91ef602209c7, 9ed7062da196487fb4b98d621ed5cdae; no voters:
I20260926 14:50:07.368611 6808 raft_consensus.cc:2781] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:07.368683 6808 raft_consensus.cc:482] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:07.368700 6808 raft_consensus.cc:3037] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Advancing to term 1
W20260926 14:50:07.368863 6638 tablet.cc:2369] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:07.369645 6808 raft_consensus.cc:504] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.369832 6808 leader_election.cc:288] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [CANDIDATE]: Term 1 election: Requested vote from peers e09c9092e0c7449085f3e8316c101294 (127.5.15.131:37769), 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.369925 6731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3940b1b4bd45038379d4284ed8c0f7" candidate_uuid: "9ed7062da196487fb4b98d621ed5cdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09c9092e0c7449085f3e8316c101294"
I20260926 14:50:07.369948 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb3940b1b4bd45038379d4284ed8c0f7" candidate_uuid: "9ed7062da196487fb4b98d621ed5cdae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e02f50502d4025a6ff91ef602209c7"
I20260926 14:50:07.370007 6591 raft_consensus.cc:3037] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.370007 6731 raft_consensus.cc:3037] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.370846 6591 raft_consensus.cc:2454] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed7062da196487fb4b98d621ed5cdae in term 1.
I20260926 14:50:07.370846 6731 raft_consensus.cc:2454] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed7062da196487fb4b98d621ed5cdae in term 1.
I20260926 14:50:07.370987 6383 leader_election.cc:302] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e02f50502d4025a6ff91ef602209c7, 9ed7062da196487fb4b98d621ed5cdae; no voters:
I20260926 14:50:07.371119 6808 raft_consensus.cc:2781] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:07.371238 6808 raft_consensus.cc:686] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 1 LEADER]: Becoming Leader. State: Replica: 9ed7062da196487fb4b98d621ed5cdae, State: Running, Role: LEADER
I20260926 14:50:07.371327 6808 consensus_queue.cc:237] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } }
I20260926 14:50:07.372134 6293 catalog_manager.cc:5495] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129). New cstate: current_term: 1 leader_uuid: "9ed7062da196487fb4b98d621ed5cdae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:07.380023 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.377449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.5.15.129:35053)
0926 14:50:07.377509 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.377511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.377558 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:50:07.377672 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.377672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.377686 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.377751 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:07.377890 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.378303 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.379199 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.379385 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.379454 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.379670 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:07.379759 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:07.379859 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.379912 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.379929 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:07.381496 6731 raft_consensus.cc:1261] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Refusing update from remote peer 9ed7062da196487fb4b98d621ed5cdae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:07.381517 6591 raft_consensus.cc:1261] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Refusing update from remote peer 9ed7062da196487fb4b98d621ed5cdae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:07.381796 6808 consensus_queue.cc:1035] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:07.381907 6810 consensus_queue.cc:1035] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:07.384019 6813 mvcc.cc:205] Tried to move back new op lower bound from 7333618513432199168 to 7333618513393524736. Current Snapshot: MvccSnapshot[applied={T|T < 7333618513432199168}]
I20260926 14:50:07.384287 6814 mvcc.cc:205] Tried to move back new op lower bound from 7333618513432199168 to 7333618513393524736. Current Snapshot: MvccSnapshot[applied={T|T < 7333618513432199168}]
I20260926 14:50:07.387449 6293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:07.387529 6293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:07.388267 6293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: Sending DeleteTablet for 3 replicas of tablet eb3940b1b4bd45038379d4284ed8c0f7
I20260926 14:50:07.388579 6431 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3940b1b4bd45038379d4284ed8c0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:07 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:46938
I20260926 14:50:07.388577 6711 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3940b1b4bd45038379d4284ed8c0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:07 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:43120
I20260926 14:50:07.388659 6571 tablet_service.cc:1514] Processing DeleteTablet for tablet eb3940b1b4bd45038379d4284ed8c0f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:07 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:58076
I20260926 14:50:07.388880 6823 tablet_replica.cc:330] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: stopping tablet replica
I20260926 14:50:07.388969 6823 raft_consensus.cc:2229] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:07.389086 6823 raft_consensus.cc:2258] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:07.389094 6293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:07.389230 6293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:07.389431 6824 tablet_replica.cc:330] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: stopping tablet replica
I20260926 14:50:07.389570 6824 raft_consensus.cc:2229] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:07.389726 6824 raft_consensus.cc:2258] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:07.389923 6823 ts_tablet_manager.cc:1905] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:07.390388 6431 tablet_service.cc:1467] Processing CreateTablet for tablet f021c7dc0fff45b68e60a8d71a249ff8 (DEFAULT_TABLE table=test-table [id=fe3334b404904c05b45ca5febc144ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:07.390409 6571 tablet_service.cc:1467] Processing CreateTablet for tablet f021c7dc0fff45b68e60a8d71a249ff8 (DEFAULT_TABLE table=test-table [id=fe3334b404904c05b45ca5febc144ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:07.390488 6431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f021c7dc0fff45b68e60a8d71a249ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.390496 6571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f021c7dc0fff45b68e60a8d71a249ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.390857 6711 tablet_service.cc:1467] Processing CreateTablet for tablet f021c7dc0fff45b68e60a8d71a249ff8 (DEFAULT_TABLE table=test-table [id=fe3334b404904c05b45ca5febc144ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:07.390936 6711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f021c7dc0fff45b68e60a8d71a249ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.391769 6800 tablet_bootstrap.cc:492] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Bootstrap starting.
I20260926 14:50:07.391929 6805 tablet_bootstrap.cc:492] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Bootstrap starting.
I20260926 14:50:07.392031 6801 tablet_bootstrap.cc:492] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Bootstrap starting.
I20260926 14:50:07.392217 6800 tablet_bootstrap.cc:654] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.392324 6805 tablet_bootstrap.cc:654] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.392431 6801 tablet_bootstrap.cc:654] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.392872 6824 ts_tablet_manager.cc:1905] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:07.392931 6805 tablet_bootstrap.cc:492] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: No bootstrap required, opened a new log
I20260926 14:50:07.392985 6805 ts_tablet_manager.cc:1397] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:07.393121 6805 raft_consensus.cc:348] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.393186 6805 raft_consensus.cc:374] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.393214 6805 raft_consensus.cc:729] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed7062da196487fb4b98d621ed5cdae, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.393253 6805 consensus_queue.cc:260] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.393330 6805 ts_tablet_manager.cc:1428] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:07.393997 6823 ts_tablet_manager.cc:1918] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:07.394085 6823 log.cc:1196] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/eb3940b1b4bd45038379d4284ed8c0f7
I20260926 14:50:07.394394 6823 ts_tablet_manager.cc:1939] T eb3940b1b4bd45038379d4284ed8c0f7 P 9ed7062da196487fb4b98d621ed5cdae: Deleting consensus metadata
I20260926 14:50:07.394949 6280 catalog_manager.cc:4841] TS 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053): tablet eb3940b1b4bd45038379d4284ed8c0f7 (table test-table [id=0d784a840cf1446badd2020487aafda2]) successfully deleted
I20260926 14:50:07.395201 6824 ts_tablet_manager.cc:1918] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:07.395293 6824 log.cc:1196] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/eb3940b1b4bd45038379d4284ed8c0f7
I20260926 14:50:07.395505 6824 ts_tablet_manager.cc:1939] T eb3940b1b4bd45038379d4284ed8c0f7 P e09c9092e0c7449085f3e8316c101294: Deleting consensus metadata
I20260926 14:50:07.395870 6825 tablet_replica.cc:330] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: stopping tablet replica
I20260926 14:50:07.395903 6281 catalog_manager.cc:4841] TS e09c9092e0c7449085f3e8316c101294 (127.5.15.131:37769): tablet eb3940b1b4bd45038379d4284ed8c0f7 (table test-table [id=0d784a840cf1446badd2020487aafda2]) successfully deleted
I20260926 14:50:07.395960 6825 raft_consensus.cc:2229] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:07.396063 6825 raft_consensus.cc:2258] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:07.396428 6825 ts_tablet_manager.cc:1905] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:07.397701 6825 ts_tablet_manager.cc:1918] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:07.397755 6825 log.cc:1196] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/eb3940b1b4bd45038379d4284ed8c0f7
I20260926 14:50:07.397994 6825 ts_tablet_manager.cc:1939] T eb3940b1b4bd45038379d4284ed8c0f7 P 49e02f50502d4025a6ff91ef602209c7: Deleting consensus metadata
I20260926 14:50:07.398375 6279 catalog_manager.cc:4841] TS 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235): tablet eb3940b1b4bd45038379d4284ed8c0f7 (table test-table [id=0d784a840cf1446badd2020487aafda2]) successfully deleted
I20260926 14:50:07.399111 6801 tablet_bootstrap.cc:492] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: No bootstrap required, opened a new log
I20260926 14:50:07.399165 6801 ts_tablet_manager.cc:1397] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260926 14:50:07.399317 6801 raft_consensus.cc:348] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.399405 6801 raft_consensus.cc:374] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.399430 6801 raft_consensus.cc:729] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e02f50502d4025a6ff91ef602209c7, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.399430 6800 tablet_bootstrap.cc:492] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: No bootstrap required, opened a new log
I20260926 14:50:07.399474 6801 consensus_queue.cc:260] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.399513 6800 ts_tablet_manager.cc:1397] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260926 14:50:07.399562 6801 ts_tablet_manager.cc:1428] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.399801 6800 raft_consensus.cc:348] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.399880 6800 raft_consensus.cc:374] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.399904 6800 raft_consensus.cc:729] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09c9092e0c7449085f3e8316c101294, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.399945 6800 consensus_queue.cc:260] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.400050 6800 ts_tablet_manager.cc:1428] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260926 14:50:07.468379 6498 tablet.cc:2369] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:07.519168 6778 tablet.cc:2369] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:07.573766 6806 raft_consensus.cc:482] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:07.573902 6806 raft_consensus.cc:504] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.574275 6806 leader_election.cc:288] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053), 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.577239 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.574581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:57325 (local address 127.5.15.130:43235)
0926 14:50:07.574639 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.574641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.574650 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:07.574676 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.574677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.574692 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.574776 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.574944 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.575487 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.576476 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.576878 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.576962 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.576972 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.577113 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.577163 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:50:07.577296 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.574484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:55799 (local address 127.5.15.129:35053)
0926 14:50:07.574534 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.574536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.574575 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:50:07.574606 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.574607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.574618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.574689 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:07.574905 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.575412 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.576545 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.576913 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.576993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.577002 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.577197 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.577236 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:50:07.577277 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.574449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:43235 (local address 127.5.15.131:57325)
0926 14:50:07.574561 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.574571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.574615 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.574775 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.574776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.574777 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.574867 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.574869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.575549 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.575550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.576372 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.576374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.576976 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.576978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.577039 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.577042 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.577109 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.577147 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260926 14:50:07.577280 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.574420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:35053 (local address 127.5.15.131:55799)
0926 14:50:07.574472 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.574485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.574524 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.574711 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.574712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.574714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:07.574821 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.574823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.575504 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.575505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.576449 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.576452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.577005 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.577006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.577076 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.577081 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.577152 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.577195 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:50:07.577718 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021c7dc0fff45b68e60a8d71a249ff8" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e02f50502d4025a6ff91ef602209c7" is_pre_election: true
I20260926 14:50:07.577826 6591 raft_consensus.cc:2454] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 0.
I20260926 14:50:07.577868 6451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021c7dc0fff45b68e60a8d71a249ff8" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed7062da196487fb4b98d621ed5cdae" is_pre_election: true
I20260926 14:50:07.578014 6451 raft_consensus.cc:2454] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 0.
I20260926 14:50:07.578037 6663 leader_election.cc:302] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e02f50502d4025a6ff91ef602209c7, e09c9092e0c7449085f3e8316c101294; no voters:
I20260926 14:50:07.578171 6806 raft_consensus.cc:2781] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:07.578220 6806 raft_consensus.cc:482] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:07.578248 6806 raft_consensus.cc:3037] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.578923 6806 raft_consensus.cc:504] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.579059 6806 leader_election.cc:288] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 election: Requested vote from peers 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053), 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.579210 6451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021c7dc0fff45b68e60a8d71a249ff8" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed7062da196487fb4b98d621ed5cdae"
I20260926 14:50:07.579270 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021c7dc0fff45b68e60a8d71a249ff8" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e02f50502d4025a6ff91ef602209c7"
I20260926 14:50:07.579309 6451 raft_consensus.cc:3037] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.579324 6591 raft_consensus.cc:3037] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.579923 6591 raft_consensus.cc:2454] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 1.
I20260926 14:50:07.580024 6451 raft_consensus.cc:2454] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 1.
I20260926 14:50:07.580106 6663 leader_election.cc:302] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e02f50502d4025a6ff91ef602209c7, e09c9092e0c7449085f3e8316c101294; no voters:
I20260926 14:50:07.580209 6806 raft_consensus.cc:2781] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:07.580397 6806 raft_consensus.cc:686] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 1 LEADER]: Becoming Leader. State: Replica: e09c9092e0c7449085f3e8316c101294, State: Running, Role: LEADER
I20260926 14:50:07.580492 6806 consensus_queue.cc:237] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.581192 6293 catalog_manager.cc:5495] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 reported cstate change: term changed from 0 to 1, leader changed from <none> to e09c9092e0c7449085f3e8316c101294 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } health_report { overall_health: HEALTHY } } }
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434206, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:07.645918 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.639431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40527 (local address 127.0.0.1:38876)
0926 14:50:07.639627 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.639636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.639715 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.639868 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.639869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.640180 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:07.640221 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.640224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.640911 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.640912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.641420 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.641421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.641746 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.641746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.641796 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.642248 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:07.642253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:07.643738 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:07.645024 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.645030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.645032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.645219 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.645387 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:07.645388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:07.645388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:07.645440 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:07.645632 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:07.645636 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:07.645676 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.645725 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.645827 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":116,"threads_started":1}
I20260926 14:50:07.646004 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.639494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.5.15.190:40527)
0926 14:50:07.639570 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.639573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.639695 (+ 122us) server_negotiation.cc:407] Connection header received
0926 14:50:07.639774 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.639775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.639788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.639864 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.640311 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.640820 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.641495 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.641666 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.641746 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.643837 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.643838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.643839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.643844 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.644959 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.645282 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.645283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.645283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.645332 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.645491 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.645492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.645492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.645582 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.645629 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.645741 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.645868 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":32}
I20260926 14:50:07.647883 6293 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:38876:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:50:07.648064 6293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:07.649248 6431 tablet_service.cc:1467] Processing CreateTablet for tablet ad16e9897a254f509dece8cbef563473 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d54c19c1721f4c5a842f948d09af0791]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:07.649322 6711 tablet_service.cc:1467] Processing CreateTablet for tablet ad16e9897a254f509dece8cbef563473 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d54c19c1721f4c5a842f948d09af0791]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:07.649317 6571 tablet_service.cc:1467] Processing CreateTablet for tablet ad16e9897a254f509dece8cbef563473 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d54c19c1721f4c5a842f948d09af0791]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:07.649428 6431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad16e9897a254f509dece8cbef563473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.649459 6571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad16e9897a254f509dece8cbef563473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.649458 6711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad16e9897a254f509dece8cbef563473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:07.650818 6805 tablet_bootstrap.cc:492] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: Bootstrap starting.
I20260926 14:50:07.650923 6801 tablet_bootstrap.cc:492] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: Bootstrap starting.
I20260926 14:50:07.651130 6805 tablet_bootstrap.cc:654] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.651315 6801 tablet_bootstrap.cc:654] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.651602 6800 tablet_bootstrap.cc:492] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: Bootstrap starting.
I20260926 14:50:07.651960 6801 tablet_bootstrap.cc:492] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: No bootstrap required, opened a new log
I20260926 14:50:07.651966 6800 tablet_bootstrap.cc:654] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:07.651999 6801 ts_tablet_manager.cc:1397] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:07.652159 6801 raft_consensus.cc:348] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.652244 6801 raft_consensus.cc:374] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.652281 6801 raft_consensus.cc:729] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e02f50502d4025a6ff91ef602209c7, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.652335 6801 consensus_queue.cc:260] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.652446 6801 ts_tablet_manager.cc:1428] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:07.652570 6804 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.652663 6804 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.652714 6800 tablet_bootstrap.cc:492] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: No bootstrap required, opened a new log
I20260926 14:50:07.652750 6800 ts_tablet_manager.cc:1397] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:07.652832 6805 tablet_bootstrap.cc:492] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: No bootstrap required, opened a new log
I20260926 14:50:07.652865 6805 ts_tablet_manager.cc:1397] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:07.652904 6800 raft_consensus.cc:348] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.652976 6800 raft_consensus.cc:374] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.652995 6800 raft_consensus.cc:729] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09c9092e0c7449085f3e8316c101294, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.652963 6805 raft_consensus.cc:348] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.653038 6805 raft_consensus.cc:374] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:07.653064 6805 raft_consensus.cc:729] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed7062da196487fb4b98d621ed5cdae, State: Initialized, Role: FOLLOWER
I20260926 14:50:07.653043 6800 consensus_queue.cc:260] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.653106 6805 consensus_queue.cc:260] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.653146 6800 ts_tablet_manager.cc:1428] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:07.653205 6805 ts_tablet_manager.cc:1428] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:07.653211 6806 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.653298 6806 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.653332 6820 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.653438 6820 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.740984 6806 raft_consensus.cc:482] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:07.741078 6806 raft_consensus.cc:504] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.741289 6806 leader_election.cc:288] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053), 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.741466 6451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad16e9897a254f509dece8cbef563473" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed7062da196487fb4b98d621ed5cdae" is_pre_election: true
I20260926 14:50:07.741482 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad16e9897a254f509dece8cbef563473" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e02f50502d4025a6ff91ef602209c7" is_pre_election: true
I20260926 14:50:07.741585 6451 raft_consensus.cc:2454] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 0.
I20260926 14:50:07.741601 6591 raft_consensus.cc:2454] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 0.
I20260926 14:50:07.742043 6664 leader_election.cc:302] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed7062da196487fb4b98d621ed5cdae, e09c9092e0c7449085f3e8316c101294; no voters:
I20260926 14:50:07.742147 6806 raft_consensus.cc:2781] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:07.742187 6806 raft_consensus.cc:482] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:07.742206 6806 raft_consensus.cc:3037] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.742904 6806 raft_consensus.cc:504] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.743045 6806 leader_election.cc:288] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 election: Requested vote from peers 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053), 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235)
I20260926 14:50:07.743186 6451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad16e9897a254f509dece8cbef563473" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed7062da196487fb4b98d621ed5cdae"
I20260926 14:50:07.743256 6451 raft_consensus.cc:3037] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.743237 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad16e9897a254f509dece8cbef563473" candidate_uuid: "e09c9092e0c7449085f3e8316c101294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e02f50502d4025a6ff91ef602209c7"
I20260926 14:50:07.743312 6591 raft_consensus.cc:3037] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:07.743988 6451 raft_consensus.cc:2454] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 1.
I20260926 14:50:07.744009 6591 raft_consensus.cc:2454] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09c9092e0c7449085f3e8316c101294 in term 1.
I20260926 14:50:07.744133 6663 leader_election.cc:302] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e02f50502d4025a6ff91ef602209c7, e09c9092e0c7449085f3e8316c101294; no voters:
I20260926 14:50:07.744234 6806 raft_consensus.cc:2781] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:07.744299 6806 raft_consensus.cc:686] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [term 1 LEADER]: Becoming Leader. State: Replica: e09c9092e0c7449085f3e8316c101294, State: Running, Role: LEADER
I20260926 14:50:07.744347 6806 consensus_queue.cc:237] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } }
I20260926 14:50:07.744652 6806 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: TxnStatusTablet state changed. Reason: New leader e09c9092e0c7449085f3e8316c101294. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.744721 6806 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:07.744895 6842 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:50:07.745009 6842 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:50:07.744956 6293 catalog_manager.cc:5495] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 reported cstate change: term changed from 0 to 1, leader changed from <none> to e09c9092e0c7449085f3e8316c101294 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:07.769553 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.766927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.5.15.131:37769)
0926 14:50:07.766968 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.766970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.766982 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:07.767090 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.767091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.767109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.767183 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:07.767307 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.767999 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.768693 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.768988 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.769059 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.769250 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:07.769348 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:07.769442 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.769466 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.769485 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:07.769568 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.766872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37769 (local address 127.0.0.1:43128)
0926 14:50:07.766942 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.766955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.767004 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.767190 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.767191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.767192 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:07.767246 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.767248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.768080 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.768081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.768616 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.768617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.769076 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.769077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.769141 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.769156 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:07.769411 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:07.769419 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.769451 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.769496 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:50:07.770568 6591 raft_consensus.cc:1261] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Refusing update from remote peer e09c9092e0c7449085f3e8316c101294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:07.770650 6451 raft_consensus.cc:1261] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Refusing update from remote peer e09c9092e0c7449085f3e8316c101294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:07.771037 6830 consensus_queue.cc:1035] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:07.771245 6806 consensus_queue.cc:1035] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:07.771660 6804 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: TxnStatusTablet state changed. Reason: New leader e09c9092e0c7449085f3e8316c101294. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.771747 6804 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.771816 6820 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: TxnStatusTablet state changed. Reason: New leader e09c9092e0c7449085f3e8316c101294. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.771896 6820 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.772341 6806 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.772420 6806 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:07.772822 6830 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.772907 6830 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P e09c9092e0c7449085f3e8316c101294: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:07.773134 6820 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.773207 6820 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P 9ed7062da196487fb4b98d621ed5cdae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.773257 6804 tablet_replica.cc:439] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e09c9092e0c7449085f3e8316c101294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 } } peers { permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 } } peers { permanent_uuid: "e09c9092e0c7449085f3e8316c101294" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37769 } } }
I20260926 14:50:07.773372 6804 tablet_replica.cc:442] T ad16e9897a254f509dece8cbef563473 P 49e02f50502d4025a6ff91ef602209c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:07.773378 6815 mvcc.cc:205] Tried to move back new op lower bound from 7333618515026530304 to 7333618514921357312. Current Snapshot: MvccSnapshot[applied={T|T < 7333618515026530304}]
I20260926 14:50:07.778184 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.775489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.5.15.131:37769)
0926 14:50:07.775524 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.775526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.775534 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:07.775590 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.775591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.775600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.775675 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:07.775769 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.776490 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.777583 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.777806 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.777857 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.778006 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:07.778028 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:07.778064 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.778115 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.778128 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:07.782670 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.780269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37769 (local address 127.5.15.131:52683)
0926 14:50:07.780389 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:07.780399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:07.780455 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:07.780566 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:07.780567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:07.780567 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:07.780605 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.780606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.781376 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.781376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.781897 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:07.781898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.782339 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.782340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:07.782381 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.782391 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:07.782546 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:07.782548 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:07.782575 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 14:50:07.782599 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":75,"threads_started":1}
I20260926 14:50:07.782673 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.780319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:52683 (local address 127.5.15.131:37769)
0926 14:50:07.780353 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.780355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.780456 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:50:07.780492 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.780492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.780500 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.780550 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:07.780642 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.781316 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.781971 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.782251 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.782322 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.782470 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:07.782495 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:07.782542 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.782589 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.782603 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:07.784637 6591 raft_consensus.cc:1261] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Refusing update from remote peer e09c9092e0c7449085f3e8316c101294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:07.784646 6451 raft_consensus.cc:1261] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Refusing update from remote peer e09c9092e0c7449085f3e8316c101294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:07.784833 6850 consensus_queue.cc:1035] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed7062da196487fb4b98d621ed5cdae" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:07.784907 6806 consensus_queue.cc:1035] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e02f50502d4025a6ff91ef602209c7" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:07.796115 6293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:07.796219 6293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:07.796770 6293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 814141e15bc144fe8fd96a1c519de046: Sending DeleteTablet for 3 replicas of tablet f021c7dc0fff45b68e60a8d71a249ff8
I20260926 14:50:07.796994 6571 tablet_service.cc:1514] Processing DeleteTablet for tablet f021c7dc0fff45b68e60a8d71a249ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:07 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:58076
I20260926 14:50:07.797013 6431 tablet_service.cc:1514] Processing DeleteTablet for tablet f021c7dc0fff45b68e60a8d71a249ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:07 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:46938
I20260926 14:50:07.797096 6825 tablet_replica.cc:330] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: stopping tablet replica
I20260926 14:50:07.797104 6711 tablet_service.cc:1514] Processing DeleteTablet for tablet f021c7dc0fff45b68e60a8d71a249ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:07 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:43120
I20260926 14:50:07.797143 6825 raft_consensus.cc:2229] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:07.797175 6825 raft_consensus.cc:2258] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:07.797192 6824 tablet_replica.cc:330] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: stopping tablet replica
I20260926 14:50:07.797233 6823 tablet_replica.cc:330] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: stopping tablet replica
I20260926 14:50:07.797248 6824 raft_consensus.cc:2229] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:07.797278 6823 raft_consensus.cc:2229] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:07.797317 6823 raft_consensus.cc:2258] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:07.797336 6824 raft_consensus.cc:2258] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:07.797639 6823 ts_tablet_manager.cc:1905] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:07.797732 6824 ts_tablet_manager.cc:1905] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:07.799037 6825 ts_tablet_manager.cc:1905] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:07.799309 6824 ts_tablet_manager.cc:1918] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:07.799381 6824 log.cc:1196] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/f021c7dc0fff45b68e60a8d71a249ff8
I20260926 14:50:07.799611 6824 ts_tablet_manager.cc:1939] T f021c7dc0fff45b68e60a8d71a249ff8 P e09c9092e0c7449085f3e8316c101294: Deleting consensus metadata
I20260926 14:50:07.799925 6823 ts_tablet_manager.cc:1918] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:07.799940 6281 catalog_manager.cc:4841] TS e09c9092e0c7449085f3e8316c101294 (127.5.15.131:37769): tablet f021c7dc0fff45b68e60a8d71a249ff8 (table test-table [id=fe3334b404904c05b45ca5febc144ddb]) successfully deleted
I20260926 14:50:07.799976 6823 log.cc:1196] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/f021c7dc0fff45b68e60a8d71a249ff8
I20260926 14:50:07.800166 6823 ts_tablet_manager.cc:1939] T f021c7dc0fff45b68e60a8d71a249ff8 P 9ed7062da196487fb4b98d621ed5cdae: Deleting consensus metadata
I20260926 14:50:07.800485 6825 ts_tablet_manager.cc:1918] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:07.800478 6280 catalog_manager.cc:4841] TS 9ed7062da196487fb4b98d621ed5cdae (127.5.15.129:35053): tablet f021c7dc0fff45b68e60a8d71a249ff8 (table test-table [id=fe3334b404904c05b45ca5febc144ddb]) successfully deleted
I20260926 14:50:07.800568 6825 log.cc:1196] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/f021c7dc0fff45b68e60a8d71a249ff8
I20260926 14:50:07.800770 6825 ts_tablet_manager.cc:1939] T f021c7dc0fff45b68e60a8d71a249ff8 P 49e02f50502d4025a6ff91ef602209c7: Deleting consensus metadata
I20260926 14:50:07.801033 6279 catalog_manager.cc:4841] TS 49e02f50502d4025a6ff91ef602209c7 (127.5.15.130:43235): tablet f021c7dc0fff45b68e60a8d71a249ff8 (table test-table [id=fe3334b404904c05b45ca5febc144ddb]) successfully deleted
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:07.810410 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.797816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.5.15.129:35053)
0926 14:50:07.797851 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.797853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.802910 (+ 5057us) server_negotiation.cc:407] Connection header received
0926 14:50:07.802970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.802970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.802980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.803034 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.803543 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.803987 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.804688 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.804893 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.804972 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.807779 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.807787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.807787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.807794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.809313 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.809640 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.809643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.809644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.809704 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.809908 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.809909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.809910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.810039 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.810192 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.810230 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.810319 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 14:50:07.810623 6491 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46960
I20260926 14:50:07.817066 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:07.811495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.5.15.190:40527)
0926 14:50:07.811553 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:07.811555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:07.811610 (+ 55us) server_negotiation.cc:407] Connection header received
0926 14:50:07.811685 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:07.811686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:07.811695 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:07.811742 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:07.812215 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.812675 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.813344 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:07.813546 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:07.813625 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:07.814680 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:07.814682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:07.814682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:07.814689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:07.815995 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.816319 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.816320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.816320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.816372 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:07.816573 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:07.816574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:07.816574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:07.816685 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:07.816781 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:07.816905 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 14:50:07.816975 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 14:50:07.817420 6293 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38882
I20260926 14:50:07.818042 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6366
I20260926 14:50:07.826988 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6506
I20260926 14:50:07.831458 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6646
I20260926 14:50:07.836300 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6262
2026-09-26T14:50:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2318 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6861](info): set up 2 sockets
Sep 26 14:50:07 dist-test-slave-s4p1 krb5kdc[6861](info): commencing operation
krb5kdc: starting...
W20260926 14:50:08.842309 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:08 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:08.950049 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:42735
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:42735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:09.026559 6874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:09.026770 6874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:09.026794 6874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:09.027920 6874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:09.027959 6874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:50:09.027972 6874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:09.027985 6874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:09.027997 6874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:09.028043 6874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:09.029474 6874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:42735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:09.029695 6874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:09.029947 6874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:09.034937 6880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.034955 6879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.035351 6874 server_base.cc:1034] running on GCE node
W20260926 14:50:09.035704 6882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.036028 6874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:09.036358 6874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:09.037505 6874 hybrid_clock.cc:648] HybridClock initialized: now 1790434209037494 us; error 28 us; skew 500 ppm
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:09.039999 6874 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:09.040342 6874 webserver.cc:460] Webserver started at http://127.5.15.190:44955/ using document root <none> and password file <none>
I20260926 14:50:09.040503 6874 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:09.040539 6874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:09.040618 6874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:09.041524 6874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "00efa4cafa1c46989ae4a20ef9ae66f4"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "a3463091edd758751f9a517331da1213"
server_key_iv: "5921abc39ffd22341ccff9345e544c2f"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.041808 6874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "00efa4cafa1c46989ae4a20ef9ae66f4"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "a3463091edd758751f9a517331da1213"
server_key_iv: "5921abc39ffd22341ccff9345e544c2f"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.043018 6874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:09.043684 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.043854 6874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.043921 6874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "00efa4cafa1c46989ae4a20ef9ae66f4"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "a3463091edd758751f9a517331da1213"
server_key_iv: "5921abc39ffd22341ccff9345e544c2f"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.044008 6874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:09.053974 6874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:09.056314 6874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:09.056468 6874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:09.060827 6874 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:42735
I20260926 14:50:09.060868 6951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:42735 every 8 connection(s)
I20260926 14:50:09.061151 6874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:09.061725 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.063853 6952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Bootstrap starting.
I20260926 14:50:09.064100 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6874
I20260926 14:50:09.064169 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:09.064330 5182 external_mini_cluster.cc:1428] Setting key 896c1abbc7fd725f35b07b591bf03839
I20260926 14:50:09.064512 6952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.064837 6952 log.cc:824] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:09.065506 6952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: No bootstrap required, opened a new log
I20260926 14:50:09.067263 6952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42735 } }
I20260926 14:50:09.067425 6952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.067471 6952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00efa4cafa1c46989ae4a20ef9ae66f4, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.067587 6952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42735 } }
I20260926 14:50:09.067675 6952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:09.067716 6952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:09.067866 6952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.068497 6952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42735 } }
I20260926 14:50:09.068598 6952 leader_election.cc:302] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00efa4cafa1c46989ae4a20ef9ae66f4; no voters:
I20260926 14:50:09.068742 6952 leader_election.cc:288] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:09.068810 6957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:09.068950 6957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [term 1 LEADER]: Becoming Leader. State: Replica: 00efa4cafa1c46989ae4a20ef9ae66f4, State: Running, Role: LEADER
I20260926 14:50:09.068979 6952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:09.069047 6957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42735 } }
I20260926 14:50:09.069363 6957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00efa4cafa1c46989ae4a20ef9ae66f4. Latest consensus state: current_term: 1 leader_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42735 } } }
I20260926 14:50:09.069454 6957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:09.069375 6959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00efa4cafa1c46989ae4a20ef9ae66f4" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42735 } } }
I20260926 14:50:09.069582 6959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:09.070815 6972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:09.070894 6972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:09.070972 6964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:09.071130 6964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:09.072501 6964 catalog_manager.cc:1347] Generated new cluster ID: 1c82a92577224a2fb69b54e449c2306a
I20260926 14:50:09.072547 6964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:09.075820 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.064803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.5.15.190:42735)
0926 14:50:09.064977 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.064985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.065017 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:09.065538 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.065545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.065614 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.065748 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.066463 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.067853 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.068947 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.069579 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.069670 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.072363 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.072367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.072367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.072378 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.074210 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.074616 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.074618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.074619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.074727 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.074959 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.074960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.074960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.075227 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.075370 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.075514 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.075634 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260926 14:50:09.092828 6964 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:09.093310 6964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:09.097882 6964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Generated new TSK 0
I20260926 14:50:09.098035 6964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:09.125891 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:42735
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:09.203696 6978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:09.203929 6978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:09.203966 6978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:09.205096 6978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:09.205160 6978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:09.205193 6978 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:50:09.205224 6978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:09.205255 6978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:09.206614 6978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:42735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:09.206888 6978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:09.207114 6978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:09.211793 6986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.211773 6983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.211766 6984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.212212 6978 server_base.cc:1034] running on GCE node
I20260926 14:50:09.212417 6978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:09.212635 6978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:09.213773 6978 hybrid_clock.cc:648] HybridClock initialized: now 1790434209213767 us; error 42 us; skew 500 ppm
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:09.216199 6978 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:09.216617 6978 webserver.cc:460] Webserver started at http://127.5.15.129:35291/ using document root <none> and password file <none>
I20260926 14:50:09.216790 6978 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:09.216830 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:09.216913 6978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:09.217846 6978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "b3818094b7f74482b1afa7b92b736016"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "53838b8311923f0a99f87b30f41987fa"
server_key_iv: "7b403a62b86926e4d5e7231d789a1aef"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.218128 6978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "b3818094b7f74482b1afa7b92b736016"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "53838b8311923f0a99f87b30f41987fa"
server_key_iv: "7b403a62b86926e4d5e7231d789a1aef"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.219403 6978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 14:50:09.220049 6993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.220252 6978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:09.220338 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "b3818094b7f74482b1afa7b92b736016"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "53838b8311923f0a99f87b30f41987fa"
server_key_iv: "7b403a62b86926e4d5e7231d789a1aef"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.220391 6978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:09.247103 6978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:09.249255 6978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:09.249359 6978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:09.249806 6978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:09.249850 6978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.249872 6978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:09.249886 6978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.256678 6978 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:44623
I20260926 14:50:09.256765 7108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:44623 every 8 connection(s)
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:09.257059 6978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:09.260049 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.250283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:41311 (local address 127.5.15.190:42735)
0926 14:50:09.250349 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.250352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.250543 (+ 191us) server_negotiation.cc:407] Connection header received
0926 14:50:09.251107 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.251108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.251119 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.251180 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.251960 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.252742 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.254362 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.254686 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.254823 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.257471 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.257472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.257473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.257480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.258988 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.259335 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.259335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.259336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.259382 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.259584 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.259585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.259585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.259676 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.259723 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.259932 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.259990 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:09.260238 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.250269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.5.15.129:41311)
0926 14:50:09.250432 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.250469 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.251009 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.251245 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.251253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.251606 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.251877 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.251884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.252847 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.252848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.254260 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.254264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.254844 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.254845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.254998 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.255557 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.255566 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.257386 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.259056 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.259062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.259064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.259274 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.259445 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.259446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.259446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.259535 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.259729 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.259733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.259833 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.259922 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.260029 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260926 14:50:09.260358 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 6978
I20260926 14:50:09.260453 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:09.260622 5182 external_mini_cluster.cc:1428] Setting key 79a9a1a93bb81520b3d2511ade33add0
I20260926 14:50:09.261641 7109 heartbeater.cc:347] Connected to a master server at 127.5.15.190:42735
I20260926 14:50:09.261723 7109 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:09.261878 7109 heartbeater.cc:511] Master 127.5.15.190:42735 requested a full tablet report, sending...
I20260926 14:50:09.262332 6905 ts_manager.cc:194] Registered new tserver with Master: b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623)
I20260926 14:50:09.263113 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:41311
I20260926 14:50:09.268077 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.261922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47317 (local address 127.5.15.190:42735)
0926 14:50:09.261956 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.261958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.261966 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:09.262096 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.262097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.262106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.262158 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.263134 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.263733 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.264399 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.264597 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.264677 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.265598 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.265599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.265600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.265606 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.266977 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.267323 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.267324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.267325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.267390 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.267594 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.267594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.267595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.267700 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.267770 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.267931 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.267996 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 14:50:09.268090 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.261755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.5.15.129:47317)
0926 14:50:09.261878 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.261889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.261993 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.262822 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.262823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.263003 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.263053 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.263055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.263823 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.263824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.264318 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.264319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.264682 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.264683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.264778 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.265237 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.265241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.265510 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.267067 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.267068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.267068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.267249 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.267454 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.267455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.267455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.267516 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.267764 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.267764 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.267817 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.267895 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.267994 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:09.285125 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:42735
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:09.363940 7118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:09.364126 7118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:09.364151 7118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:09.365281 7118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:09.365321 7118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:09.365341 7118 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:50:09.365353 7118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:09.365407 7118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:09.366787 7118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:42735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:09.367051 7118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:09.367302 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:09.372120 7123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.372143 7124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.372124 7126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.372225 7118 server_base.cc:1034] running on GCE node
I20260926 14:50:09.372498 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:09.372756 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:09.373914 7118 hybrid_clock.cc:648] HybridClock initialized: now 1790434209373896 us; error 31 us; skew 500 ppm
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:09.376248 7118 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:09.376663 7118 webserver.cc:460] Webserver started at http://127.5.15.130:43949/ using document root <none> and password file <none>
I20260926 14:50:09.376999 7118 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:09.377061 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:09.377218 7118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:09.378204 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "392289e8b0f044a99d089f74a8e8865b"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "ed81912a8b0f937d1042b7e301d14836"
server_key_iv: "58b4d9384168dee91fc8f4d697f8214d"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.378558 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "392289e8b0f044a99d089f74a8e8865b"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "ed81912a8b0f937d1042b7e301d14836"
server_key_iv: "58b4d9384168dee91fc8f4d697f8214d"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.379900 7118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:09.380576 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.380755 7118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:09.380851 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "392289e8b0f044a99d089f74a8e8865b"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "ed81912a8b0f937d1042b7e301d14836"
server_key_iv: "58b4d9384168dee91fc8f4d697f8214d"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.380947 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:09.395816 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:09.397596 7118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:09.397745 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:09.398217 7118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:09.398284 7118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.398324 7118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:09.398355 7118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.404824 7118 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:32805
I20260926 14:50:09.404922 7248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:32805 every 8 connection(s)
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:09.405180 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:09.407855 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.398720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:44507 (local address 127.5.15.190:42735)
0926 14:50:09.398788 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.398790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.399030 (+ 240us) server_negotiation.cc:407] Connection header received
0926 14:50:09.399552 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.399555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.399568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.399649 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.400493 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.401303 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.402806 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.403113 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.403195 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.405748 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.405750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.405750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.405756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.406893 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.407215 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.407216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.407216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.407262 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.407417 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.407417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.407417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.407505 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.407548 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.407723 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.407795 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:09.408079 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.398726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.5.15.130:44507)
0926 14:50:09.398907 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.398939 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.399444 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.399758 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.399761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.400127 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.400391 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.400398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.401395 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.401397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.402685 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.402688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.403211 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.403211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.403308 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.403825 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.403835 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.405665 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.406955 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.406960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.406961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.407153 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.407300 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.407301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.407301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.407377 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.407539 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.407541 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.407632 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.407701 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.407831 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 14:50:09.408607 7249 heartbeater.cc:347] Connected to a master server at 127.5.15.190:42735
I20260926 14:50:09.408701 7249 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:09.408866 7249 heartbeater.cc:511] Master 127.5.15.190:42735 requested a full tablet report, sending...
I20260926 14:50:09.409224 6906 ts_manager.cc:194] Registered new tserver with Master: 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805)
I20260926 14:50:09.409593 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 7118
I20260926 14:50:09.409632 6906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:44507
I20260926 14:50:09.409693 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:09.409871 5182 external_mini_cluster.cc:1428] Setting key c7abbb00a125b9573a689dc92bfb621c
I20260926 14:50:09.415767 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.409609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:59685 (local address 127.5.15.190:42735)
0926 14:50:09.409739 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.409740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.410125 (+ 385us) server_negotiation.cc:407] Connection header received
0926 14:50:09.410219 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.410220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.410229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.410292 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.410625 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.411270 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.411974 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.412177 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.412264 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.413159 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.413160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.413160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.413167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.414550 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.414975 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.414976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.414976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.415035 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.415226 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.415235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.415235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.415372 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.415443 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.415611 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.415680 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260926 14:50:09.415791 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.409558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.5.15.130:59685)
0926 14:50:09.410066 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.410077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.410148 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.410299 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.410300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.410487 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.410552 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.410554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.411354 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.411355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.411891 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.411893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.412266 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.412267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.412347 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.412762 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.412767 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.413057 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.414630 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.414631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.414632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.414893 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.415099 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.415100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.415100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.415156 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.415448 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.415449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.415501 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.415585 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.415682 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:09.434124 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:42735
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:09.512012 7258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:09.512193 7258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:09.512217 7258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:09.513334 7258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:09.513372 7258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:09.513392 7258 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:50:09.513442 7258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:09.513473 7258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:09.514892 7258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:42735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:09.515106 7258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:09.515322 7258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:09.520058 7266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.520059 7263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:09.520081 7264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:09.520285 7258 server_base.cc:1034] running on GCE node
I20260926 14:50:09.520470 7258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:09.520663 7258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:09.521790 7258 hybrid_clock.cc:648] HybridClock initialized: now 1790434209521776 us; error 33 us; skew 500 ppm
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:09.524125 7258 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:09.524590 7258 webserver.cc:460] Webserver started at http://127.5.15.131:42915/ using document root <none> and password file <none>
I20260926 14:50:09.524840 7258 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:09.524906 7258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:09.525051 7258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:09.526012 7258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "d11f5f75747e46329be1d9b83096610e"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "6366f77da89d07ddf62603ac2d14dac6"
server_key_iv: "7b87bc9e5c923d7378158e8f5f444fcc"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.526407 7258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "d11f5f75747e46329be1d9b83096610e"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "6366f77da89d07ddf62603ac2d14dac6"
server_key_iv: "7b87bc9e5c923d7378158e8f5f444fcc"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.527737 7258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:09.528402 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.528561 7258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:09.528651 7258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "d11f5f75747e46329be1d9b83096610e"
format_stamp: "Formatted at 2026-09-26 14:50:09 on dist-test-slave-s4p1"
server_key: "6366f77da89d07ddf62603ac2d14dac6"
server_key_iv: "7b87bc9e5c923d7378158e8f5f444fcc"
server_key_version: "encryptionkey@0"
I20260926 14:50:09.528724 7258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:09.540906 7258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:09.542595 7258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:09.542786 7258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:09.543306 7258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:09.543360 7258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.543399 7258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:09.543422 7258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:09.550729 7258 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:37545
I20260926 14:50:09.550856 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:37545 every 8 connection(s)
I20260926 14:50:09.551141 7258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:09.552655 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.543652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38003 (local address 127.5.15.190:42735)
0926 14:50:09.543702 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.543704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.543928 (+ 224us) server_negotiation.cc:407] Connection header received
0926 14:50:09.544351 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.544352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.544366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.544450 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.544987 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.545480 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.546596 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.546908 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.546991 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.549921 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.549922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.549923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.549930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.551663 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.552000 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.552000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.552000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.552049 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.552208 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.552209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.552209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.552304 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.552355 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.552539 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.552596 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:50:09.552827 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.543635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.5.15.131:38003)
0926 14:50:09.543789 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.543826 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.544258 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.544493 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.544496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.544708 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.544889 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.544893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.545573 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.545574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.546510 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.546513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.546994 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.546995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.547135 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.547634 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.547645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.549823 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.551727 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.551730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.551731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.551934 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.552087 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.552088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.552088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.552161 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.552348 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.552351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.552445 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.552514 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.552616 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260926 14:50:09.553375 7389 heartbeater.cc:347] Connected to a master server at 127.5.15.190:42735
I20260926 14:50:09.553443 7389 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:09.553632 7389 heartbeater.cc:511] Master 127.5.15.190:42735 requested a full tablet report, sending...
I20260926 14:50:09.553993 6905 ts_manager.cc:194] Registered new tserver with Master: d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:09.554466 6905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:38003
I20260926 14:50:09.558228 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 7258
I20260926 14:50:09.558305 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:09.558495 5182 external_mini_cluster.cc:1428] Setting key 494cdd5782b72df7dc0c2986073ef0ec
I20260926 14:50:09.559408 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:09.559700 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.554283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.5.15.131:55479)
0926 14:50:09.554436 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.554447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.554508 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.554628 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.554629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.554899 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.554956 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.554958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.555646 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.555647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.556141 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.556143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.556441 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.556442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.556495 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.556913 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.556918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.557179 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.558517 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.558518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.558518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.558738 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.559042 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.559043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.559043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.559131 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.559406 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.559406 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.559441 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.559546 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.559632 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":105,"threads_started":1}
I20260926 14:50:09.559746 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.554362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:55479 (local address 127.5.15.190:42735)
0926 14:50:09.554420 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.554422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.554499 (+ 77us) server_negotiation.cc:407] Connection header received
0926 14:50:09.554558 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.554559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.554568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.554625 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.555041 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.555513 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.556205 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.556379 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.556429 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.557271 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.557272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.557272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.557277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.558427 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.558852 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.558853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.558853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.558915 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.559208 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.559209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.559209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.559338 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.559405 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.559602 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.559671 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:09.582082 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.575229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.5.15.190:42735)
0926 14:50:09.575290 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.575292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.575302 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:09.575354 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.575354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.575362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.575410 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.575951 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.576433 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.577172 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.577396 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.577451 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.579556 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.579557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.579557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.579566 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.580952 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.581323 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.581325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.581325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.581385 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.581579 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.581580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.581580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.581694 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.581792 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.581905 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.581979 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:09.583645 6905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:09.585367 6905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:09.591980 7405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.587512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:32805 (local address 127.0.0.1:49504)
0926 14:50:09.587693 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.587750 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.587817 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.588220 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.588220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.588222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.588268 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.588269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.589548 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.589548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.590610 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.590612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.591631 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.591632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.591711 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.591729 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.591835 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.591893 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260926 14:50:09.591990 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.587600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.5.15.130:32805)
0926 14:50:09.587955 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.587959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.587969 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:09.588003 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.588005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.588021 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.588115 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.588350 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.589463 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.590697 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.591454 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.591535 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.591595 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.591854 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.591892 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":83,"threads_started":1}
I20260926 14:50:09.592587 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.587527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37545 (local address 127.0.0.1:52926)
0926 14:50:09.587906 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.587914 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.587954 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.588427 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.588428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.588430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.588486 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.588487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.589319 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.589320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.590610 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.590612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.592322 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.592323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.592385 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.592389 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.592446 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.592497 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260926 14:50:09.592592 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.587766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.5.15.131:37545)
0926 14:50:09.588227 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.588231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.588240 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:09.588274 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.588276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.588290 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.588363 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.588568 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.589224 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.590697 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.591459 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.591534 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.591595 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.592460 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.592504 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":73,"threads_started":1}
I20260926 14:50:09.592746 7183 tablet_service.cc:1467] Processing CreateTablet for tablet fdce2ffc4c534d379085ec3aa0ebbd4c (DEFAULT_TABLE table=test-table [id=fc2349b876ed4cc5b5e4a595cde8f875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:09.593119 7183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdce2ffc4c534d379085ec3aa0ebbd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.593267 7323 tablet_service.cc:1467] Processing CreateTablet for tablet fdce2ffc4c534d379085ec3aa0ebbd4c (DEFAULT_TABLE table=test-table [id=fc2349b876ed4cc5b5e4a595cde8f875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:09.593561 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdce2ffc4c534d379085ec3aa0ebbd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.595029 7403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.587527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44623 (local address 127.0.0.1:32768)
0926 14:50:09.587755 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.587764 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.587817 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.588129 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.588132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.588135 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.588206 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.588208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.590713 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.590714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.591456 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.591458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.592235 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.592236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.594847 (+ 2611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.594856 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.594925 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.594967 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":59,"threads_started":1}
I20260926 14:50:09.595085 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.587768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.5.15.129:44623)
0926 14:50:09.587943 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.587948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.587960 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:09.588003 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.588005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.588021 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.588115 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.588315 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.589463 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.591557 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.592154 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.592229 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.592287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.594944 (+ 2657us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.594989 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260926 14:50:09.595911 7043 tablet_service.cc:1467] Processing CreateTablet for tablet fdce2ffc4c534d379085ec3aa0ebbd4c (DEFAULT_TABLE table=test-table [id=fc2349b876ed4cc5b5e4a595cde8f875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:09.596211 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdce2ffc4c534d379085ec3aa0ebbd4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.596931 7409 tablet_bootstrap.cc:492] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Bootstrap starting.
I20260926 14:50:09.597041 7410 tablet_bootstrap.cc:492] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Bootstrap starting.
I20260926 14:50:09.597551 7410 tablet_bootstrap.cc:654] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.597805 7409 tablet_bootstrap.cc:654] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.597867 7410 log.cc:824] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:09.598122 7409 log.cc:824] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:09.598830 7410 tablet_bootstrap.cc:492] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: No bootstrap required, opened a new log
I20260926 14:50:09.598897 7410 ts_tablet_manager.cc:1397] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:09.598912 7409 tablet_bootstrap.cc:492] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: No bootstrap required, opened a new log
I20260926 14:50:09.598970 7409 ts_tablet_manager.cc:1397] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:09.599957 7413 tablet_bootstrap.cc:492] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Bootstrap starting.
I20260926 14:50:09.600586 7413 tablet_bootstrap.cc:654] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.600685 7410 raft_consensus.cc:348] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.600682 7409 raft_consensus.cc:348] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.600854 7409 raft_consensus.cc:374] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.600854 7410 raft_consensus.cc:374] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.600901 7410 raft_consensus.cc:729] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392289e8b0f044a99d089f74a8e8865b, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.600901 7409 raft_consensus.cc:729] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.600907 7413 log.cc:824] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:09.601012 7410 consensus_queue.cc:260] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.601068 7409 consensus_queue.cc:260] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.601308 7410 ts_tablet_manager.cc:1428] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:09.601337 7409 ts_tablet_manager.cc:1428] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:09.601357 7389 heartbeater.cc:503] Master 127.5.15.190:42735 was elected leader, sending a full tablet report...
I20260926 14:50:09.601540 7249 heartbeater.cc:503] Master 127.5.15.190:42735 was elected leader, sending a full tablet report...
I20260926 14:50:09.601989 7413 tablet_bootstrap.cc:492] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: No bootstrap required, opened a new log
I20260926 14:50:09.602041 7413 ts_tablet_manager.cc:1397] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:09.603823 7413 raft_consensus.cc:348] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.603937 7413 raft_consensus.cc:374] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.603956 7413 raft_consensus.cc:729] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.604080 7413 consensus_queue.cc:260] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.604355 7109 heartbeater.cc:503] Master 127.5.15.190:42735 was elected leader, sending a full tablet report...
I20260926 14:50:09.604341 7413 ts_tablet_manager.cc:1428] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260926 14:50:09.655786 7250 tablet.cc:2369] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:09.694597 7416 raft_consensus.cc:482] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:09.694830 7416 raft_consensus.cc:504] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.695227 7416 leader_election.cc:288] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805), b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623)
I20260926 14:50:09.697970 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.695520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:33201 (local address 127.5.15.130:32805)
0926 14:50:09.695562 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.695564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.695576 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:09.695607 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.695608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.695631 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.695701 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.695861 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.696348 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.697247 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.697620 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.697705 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.697715 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.697868 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.697902 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:09.697969 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.695423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:32805 (local address 127.5.15.131:33201)
0926 14:50:09.695497 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.695509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.695554 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.695717 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.695718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.695719 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.695789 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.695790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.696435 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.696436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.697152 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.697153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.697703 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.697704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.697786 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.697789 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.697850 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.697891 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 14:50:09.698158 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.695549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:48361 (local address 127.5.15.129:44623)
0926 14:50:09.695593 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.695595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.695741 (+ 146us) server_negotiation.cc:407] Connection header received
0926 14:50:09.695808 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.695809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.695821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.695890 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.696072 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.696526 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.697414 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.697775 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.697865 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.697873 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.698015 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.698063 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:09.698350 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.695465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44623 (local address 127.5.15.131:48361)
0926 14:50:09.695655 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.695666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.695748 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.695904 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.695905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.695907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.695998 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.695999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.696611 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.696612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.697297 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.697299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.697863 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.697864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.697940 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.697943 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.698010 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.698193 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260926 14:50:09.698297 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdce2ffc4c534d379085ec3aa0ebbd4c" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b" is_pre_election: true
I20260926 14:50:09.698457 7203 raft_consensus.cc:2454] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 0.
I20260926 14:50:09.698702 7277 leader_election.cc:302] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392289e8b0f044a99d089f74a8e8865b, d11f5f75747e46329be1d9b83096610e; no voters:
I20260926 14:50:09.698711 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdce2ffc4c534d379085ec3aa0ebbd4c" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3818094b7f74482b1afa7b92b736016" is_pre_election: true
I20260926 14:50:09.698840 7416 raft_consensus.cc:2781] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:09.698902 7416 raft_consensus.cc:482] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:09.698897 7063 raft_consensus.cc:2454] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 0.
I20260926 14:50:09.698935 7416 raft_consensus.cc:3037] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.699644 7416 raft_consensus.cc:504] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.699797 7416 leader_election.cc:288] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 election: Requested vote from peers 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805), b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623)
I20260926 14:50:09.699958 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdce2ffc4c534d379085ec3aa0ebbd4c" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3818094b7f74482b1afa7b92b736016"
I20260926 14:50:09.699963 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdce2ffc4c534d379085ec3aa0ebbd4c" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b"
I20260926 14:50:09.700037 7063 raft_consensus.cc:3037] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.700035 7203 raft_consensus.cc:3037] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.700891 7203 raft_consensus.cc:2454] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 1.
I20260926 14:50:09.700886 7063 raft_consensus.cc:2454] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 1.
I20260926 14:50:09.701102 7277 leader_election.cc:302] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392289e8b0f044a99d089f74a8e8865b, d11f5f75747e46329be1d9b83096610e; no voters:
I20260926 14:50:09.701225 7416 raft_consensus.cc:2781] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:09.701395 7416 raft_consensus.cc:686] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 1 LEADER]: Becoming Leader. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Running, Role: LEADER
I20260926 14:50:09.701490 7416 consensus_queue.cc:237] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.702351 6905 catalog_manager.cc:5495] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e reported cstate change: term changed from 0 to 1, leader changed from <none> to d11f5f75747e46329be1d9b83096610e (127.5.15.131). New cstate: current_term: 1 leader_uuid: "d11f5f75747e46329be1d9b83096610e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } health_report { overall_health: HEALTHY } } }
W20260926 14:50:09.757848 7110 tablet.cc:2369] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 14:50:09 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:09.762284 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.755485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.5.15.190:42735)
0926 14:50:09.755571 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.755573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.755714 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:50:09.755760 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.755761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.755770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.755828 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:09.756253 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.756733 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.757443 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.757614 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.757664 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.759677 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:09.759691 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:09.759691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:09.759697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:09.761066 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.761414 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.761415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.761415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.761506 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:09.761723 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:09.761724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:09.761725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:09.761829 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:09.761902 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.762055 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.762165 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":53}
I20260926 14:50:09.762360 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.755425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:42735 (local address 127.0.0.1:48454)
0926 14:50:09.755589 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.755609 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.755669 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.755841 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.755841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.756132 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:09.756176 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.756178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.756794 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.756794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.757354 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.757355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.757676 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.757676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.757716 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.758281 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:09.758285 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:09.759597 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:09.761152 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.761158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.761159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.761351 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.761584 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:09.761585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:09.761585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:09.761648 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:09.761893 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:09.761900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:09.761945 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.762030 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.762243 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":88,"thread_start_us":104,"threads_started":1}
I20260926 14:50:09.764248 6905 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:48454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:50:09.764405 6905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:09.765761 7043 tablet_service.cc:1467] Processing CreateTablet for tablet c2b43c6f8d4a43b1a785902902e6d4f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42c91e9ca43e48caa922d782961d4f95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:09.765834 7183 tablet_service.cc:1467] Processing CreateTablet for tablet c2b43c6f8d4a43b1a785902902e6d4f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42c91e9ca43e48caa922d782961d4f95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:09.765925 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b43c6f8d4a43b1a785902902e6d4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.765928 7323 tablet_service.cc:1467] Processing CreateTablet for tablet c2b43c6f8d4a43b1a785902902e6d4f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=42c91e9ca43e48caa922d782961d4f95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:09.766009 7183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b43c6f8d4a43b1a785902902e6d4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.766059 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b43c6f8d4a43b1a785902902e6d4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.767235 7413 tablet_bootstrap.cc:492] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: Bootstrap starting.
I20260926 14:50:09.767554 7410 tablet_bootstrap.cc:492] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: Bootstrap starting.
I20260926 14:50:09.767644 7413 tablet_bootstrap.cc:654] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.767870 7409 tablet_bootstrap.cc:492] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: Bootstrap starting.
I20260926 14:50:09.768313 7410 tablet_bootstrap.cc:654] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.768327 7409 tablet_bootstrap.cc:654] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.768468 7413 tablet_bootstrap.cc:492] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: No bootstrap required, opened a new log
I20260926 14:50:09.768517 7413 ts_tablet_manager.cc:1397] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:09.768637 7413 raft_consensus.cc:348] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.768736 7413 raft_consensus.cc:374] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.768764 7413 raft_consensus.cc:729] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.768814 7413 consensus_queue.cc:260] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.768932 7413 ts_tablet_manager.cc:1428] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.768981 7417 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.769048 7410 tablet_bootstrap.cc:492] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: No bootstrap required, opened a new log
I20260926 14:50:09.769093 7410 ts_tablet_manager.cc:1397] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:09.769100 7417 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:09.769256 7410 raft_consensus.cc:348] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.769368 7410 raft_consensus.cc:374] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.769395 7410 raft_consensus.cc:729] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392289e8b0f044a99d089f74a8e8865b, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.769448 7410 consensus_queue.cc:260] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.769580 7410 ts_tablet_manager.cc:1428] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.769670 7415 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.769799 7415 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:09.769927 7409 tablet_bootstrap.cc:492] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: No bootstrap required, opened a new log
I20260926 14:50:09.769975 7409 ts_tablet_manager.cc:1397] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:09.770138 7409 raft_consensus.cc:348] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.770197 7409 raft_consensus.cc:374] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.770211 7409 raft_consensus.cc:729] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.770270 7409 consensus_queue.cc:260] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.770377 7409 ts_tablet_manager.cc:1428] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.770450 7416 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.770545 7416 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 14:50:09.801687 7390 tablet.cc:2369] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:09.823935 7415 raft_consensus.cc:482] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:09.824034 7415 raft_consensus.cc:504] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.824404 7415 leader_election.cc:288] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623), d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:09.827118 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.824629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:43023 (local address 127.5.15.129:44623)
0926 14:50:09.824670 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.824673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.824682 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:09.824746 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.824764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.824774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.824864 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.825019 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.825501 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.826434 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.826812 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.826889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.826897 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.827024 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.827061 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:09.827118 7142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.824564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44623 (local address 127.5.15.130:43023)
0926 14:50:09.824617 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.824628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.824665 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.824867 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.824868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.824870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.824942 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.824943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.825582 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.825583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.826345 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.826347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.826885 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.826886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.826958 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.826961 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.827020 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.827055 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:50:09.827194 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.824628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:35339 (local address 127.5.15.131:37545)
0926 14:50:09.824670 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.824673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.824852 (+ 179us) server_negotiation.cc:407] Connection header received
0926 14:50:09.824933 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.824934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.824945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.824984 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.825133 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.825583 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.826459 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.826882 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.826954 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.826961 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.827104 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.827138 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:09.827342 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b43c6f8d4a43b1a785902902e6d4f6" candidate_uuid: "392289e8b0f044a99d089f74a8e8865b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3818094b7f74482b1afa7b92b736016" is_pre_election: true
I20260926 14:50:09.827394 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.824576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37545 (local address 127.5.15.130:35339)
0926 14:50:09.824769 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.824779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.824864 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.824995 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.824996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.824998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.825067 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.825069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.825665 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.825666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.826370 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.826371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.826983 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.826984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.827039 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.827042 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.827100 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.827133 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260926 14:50:09.827438 7063 raft_consensus.cc:2454] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 392289e8b0f044a99d089f74a8e8865b in term 0.
I20260926 14:50:09.827625 7135 leader_election.cc:302] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392289e8b0f044a99d089f74a8e8865b, b3818094b7f74482b1afa7b92b736016; no voters:
I20260926 14:50:09.827731 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b43c6f8d4a43b1a785902902e6d4f6" candidate_uuid: "392289e8b0f044a99d089f74a8e8865b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11f5f75747e46329be1d9b83096610e" is_pre_election: true
I20260926 14:50:09.827802 7415 raft_consensus.cc:2781] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:09.827852 7415 raft_consensus.cc:482] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:09.827852 7343 raft_consensus.cc:2454] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 392289e8b0f044a99d089f74a8e8865b in term 0.
I20260926 14:50:09.827877 7415 raft_consensus.cc:3037] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.828569 7415 raft_consensus.cc:504] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.828706 7415 leader_election.cc:288] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [CANDIDATE]: Term 1 election: Requested vote from peers b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623), d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:09.828832 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b43c6f8d4a43b1a785902902e6d4f6" candidate_uuid: "392289e8b0f044a99d089f74a8e8865b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3818094b7f74482b1afa7b92b736016"
I20260926 14:50:09.828857 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b43c6f8d4a43b1a785902902e6d4f6" candidate_uuid: "392289e8b0f044a99d089f74a8e8865b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11f5f75747e46329be1d9b83096610e"
I20260926 14:50:09.828918 7063 raft_consensus.cc:3037] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.828920 7343 raft_consensus.cc:3037] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.829466 7063 raft_consensus.cc:2454] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 392289e8b0f044a99d089f74a8e8865b in term 1.
I20260926 14:50:09.829592 7343 raft_consensus.cc:2454] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 392289e8b0f044a99d089f74a8e8865b in term 1.
I20260926 14:50:09.829674 7135 leader_election.cc:302] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392289e8b0f044a99d089f74a8e8865b, b3818094b7f74482b1afa7b92b736016; no voters:
I20260926 14:50:09.829773 7415 raft_consensus.cc:2781] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:09.829919 7415 raft_consensus.cc:686] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [term 1 LEADER]: Becoming Leader. State: Replica: 392289e8b0f044a99d089f74a8e8865b, State: Running, Role: LEADER
I20260926 14:50:09.830027 7415 consensus_queue.cc:237] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.830479 7431 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: TxnStatusTablet state changed. Reason: New leader 392289e8b0f044a99d089f74a8e8865b. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.830560 7431 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:09.830734 7433 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:50:09.830852 7433 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:50:09.830837 6905 catalog_manager.cc:5495] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b reported cstate change: term changed from 0 to 1, leader changed from <none> to 392289e8b0f044a99d089f74a8e8865b (127.5.15.130). New cstate: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:09.842434 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.840044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.5.15.130:32805)
0926 14:50:09.840096 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.840097 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.840104 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:09.840153 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.840154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.840167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.840246 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:09.840389 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.840853 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.841645 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.841846 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.841921 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.842096 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.842191 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.842279 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.842331 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.842351 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:09.842448 7426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.839810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:32805 (local address 127.0.0.1:49520)
0926 14:50:09.839902 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.839913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.839962 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.840260 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.840261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.840262 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:09.840319 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.840321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.840935 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.840936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.841566 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.841568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.841920 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.841921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.841987 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.842003 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.842260 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.842269 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.842325 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.842374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 14:50:09.844209 7063 raft_consensus.cc:1261] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Refusing update from remote peer 392289e8b0f044a99d089f74a8e8865b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:09.844348 7343 raft_consensus.cc:1261] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Refusing update from remote peer 392289e8b0f044a99d089f74a8e8865b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:09.844558 7415 consensus_queue.cc:1035] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:09.844691 7431 consensus_queue.cc:1035] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:09.845767 7417 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: TxnStatusTablet state changed. Reason: New leader 392289e8b0f044a99d089f74a8e8865b. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.845860 7417 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:09.845935 7416 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: TxnStatusTablet state changed. Reason: New leader 392289e8b0f044a99d089f74a8e8865b. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.846029 7416 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:09.846980 7415 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.847064 7415 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:09.847162 7415 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.847241 7415 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P 392289e8b0f044a99d089f74a8e8865b: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:09.847764 7416 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.847847 7416 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P d11f5f75747e46329be1d9b83096610e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:09.847805 7436 mvcc.cc:205] Tried to move back new op lower bound from 7333618523519451136 to 7333618523464294400. Current Snapshot: MvccSnapshot[applied={T|T < 7333618523519451136 or (T in {7333618523519451136})}]
I20260926 14:50:09.848040 7438 mvcc.cc:205] Tried to move back new op lower bound from 7333618523519451136 to 7333618523464294400. Current Snapshot: MvccSnapshot[applied={T|T < 7333618523519451136}]
I20260926 14:50:09.848346 7417 tablet_replica.cc:439] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "392289e8b0f044a99d089f74a8e8865b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } } }
I20260926 14:50:09.848420 7417 tablet_replica.cc:442] T c2b43c6f8d4a43b1a785902902e6d4f6 P b3818094b7f74482b1afa7b92b736016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:09.848523 7437 mvcc.cc:205] Tried to move back new op lower bound from 7333618523519451136 to 7333618523464294400. Current Snapshot: MvccSnapshot[applied={T|T < 7333618523519451136}]
I20260926 14:50:09.851890 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.849321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.5.15.131:37545)
0926 14:50:09.849358 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.849360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.849415 (+ 55us) server_negotiation.cc:407] Connection header received
0926 14:50:09.849495 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.849495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.849508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.849553 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:09.849679 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.850092 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.851188 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.851381 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.851432 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.851596 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.851674 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.851730 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.851793 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.851808 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:09.856426 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.854218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:60187 (local address 127.5.15.130:32805)
0926 14:50:09.854256 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.854258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.854286 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:50:09.854353 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.854353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.854363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.854404 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:09.854528 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.855039 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.855746 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.855924 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.856004 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.856197 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.856231 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.856288 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.856345 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.856362 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:09.856442 7395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.854166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:32805 (local address 127.5.15.131:60187)
0926 14:50:09.854245 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.854255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.854299 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.854414 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.854415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.854416 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:09.854480 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.854482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.855118 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.855119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.855661 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.855663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.856005 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.856006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.856071 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.856082 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.856288 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.856291 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.856341 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.856373 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:50:09.858950 7063 raft_consensus.cc:1261] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Refusing update from remote peer d11f5f75747e46329be1d9b83096610e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:09.859010 7203 raft_consensus.cc:1261] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Refusing update from remote peer d11f5f75747e46329be1d9b83096610e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:09.859223 7419 consensus_queue.cc:1035] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:09.859366 7419 consensus_queue.cc:1035] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:09.866829 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.864410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:37955 (local address 127.5.15.131:37545)
0926 14:50:09.864449 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.864451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.864459 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:09.864515 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.864516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.864526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.864576 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:09.864681 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.865124 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.866109 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.866410 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.866475 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.866599 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.866628 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.866683 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.866733 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.866769 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:09.866864 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.864366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37545 (local address 127.5.15.130:37955)
0926 14:50:09.864410 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.864420 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.864459 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.864578 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.864579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.864580 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:09.864626 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.864627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.865203 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.865204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.866020 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.866022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.866478 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.866479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.866522 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.866533 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:09.866681 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:09.866682 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.866728 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.866806 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260926 14:50:09.875687 6905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:09.875801 6905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:09.876478 6905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Sending DeleteTablet for 3 replicas of tablet fdce2ffc4c534d379085ec3aa0ebbd4c
I20260926 14:50:09.876792 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet fdce2ffc4c534d379085ec3aa0ebbd4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:09 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:32768
I20260926 14:50:09.876832 7183 tablet_service.cc:1514] Processing DeleteTablet for tablet fdce2ffc4c534d379085ec3aa0ebbd4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:09 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:49504
I20260926 14:50:09.876870 7323 tablet_service.cc:1514] Processing DeleteTablet for tablet fdce2ffc4c534d379085ec3aa0ebbd4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:09 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:52926
I20260926 14:50:09.877079 7456 tablet_replica.cc:330] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: stopping tablet replica
I20260926 14:50:09.877163 7456 raft_consensus.cc:2229] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:09.877254 7456 raft_consensus.cc:2258] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:09.877410 7455 tablet_replica.cc:330] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: stopping tablet replica
I20260926 14:50:09.877426 6905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:50:09.877497 7455 raft_consensus.cc:2229] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:09.877537 7455 raft_consensus.cc:2258] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:50:09.877542 6905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:09.878496 7455 ts_tablet_manager.cc:1905] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:09.878866 7323 tablet_service.cc:1467] Processing CreateTablet for tablet f9820027600d48d8901c7af440f88adb (DEFAULT_TABLE table=test-table [id=b3488bf0650e4986bbb93fe61cee202f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:09.878980 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9820027600d48d8901c7af440f88adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.880321 7409 tablet_bootstrap.cc:492] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Bootstrap starting.
I20260926 14:50:09.880585 7455 ts_tablet_manager.cc:1918] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:09.880703 7455 log.cc:1196] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/fdce2ffc4c534d379085ec3aa0ebbd4c
I20260926 14:50:09.880718 7409 tablet_bootstrap.cc:654] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.881044 7455 ts_tablet_manager.cc:1939] T fdce2ffc4c534d379085ec3aa0ebbd4c P 392289e8b0f044a99d089f74a8e8865b: Deleting consensus metadata
I20260926 14:50:09.881160 7183 tablet_service.cc:1467] Processing CreateTablet for tablet f9820027600d48d8901c7af440f88adb (DEFAULT_TABLE table=test-table [id=b3488bf0650e4986bbb93fe61cee202f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:09.881253 7183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9820027600d48d8901c7af440f88adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.882221 7410 tablet_bootstrap.cc:492] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Bootstrap starting.
I20260926 14:50:09.882303 6893 catalog_manager.cc:4841] TS 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805): tablet fdce2ffc4c534d379085ec3aa0ebbd4c (table test-table [id=fc2349b876ed4cc5b5e4a595cde8f875]) successfully deleted
I20260926 14:50:09.882615 7410 tablet_bootstrap.cc:654] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.882767 7043 tablet_service.cc:1467] Processing CreateTablet for tablet f9820027600d48d8901c7af440f88adb (DEFAULT_TABLE table=test-table [id=b3488bf0650e4986bbb93fe61cee202f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:09.882874 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9820027600d48d8901c7af440f88adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:09.883251 7410 tablet_bootstrap.cc:492] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: No bootstrap required, opened a new log
I20260926 14:50:09.883293 7410 ts_tablet_manager.cc:1397] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:09.883417 7410 raft_consensus.cc:348] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.883471 7410 raft_consensus.cc:374] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.883495 7410 raft_consensus.cc:729] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392289e8b0f044a99d089f74a8e8865b, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.883533 7410 consensus_queue.cc:260] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.883615 7410 ts_tablet_manager.cc:1428] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.884249 7413 tablet_bootstrap.cc:492] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Bootstrap starting.
I20260926 14:50:09.884624 7413 tablet_bootstrap.cc:654] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:09.885264 7456 ts_tablet_manager.cc:1905] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:09.885339 7413 tablet_bootstrap.cc:492] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: No bootstrap required, opened a new log
I20260926 14:50:09.885373 7413 ts_tablet_manager.cc:1397] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:09.885505 7413 raft_consensus.cc:348] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.885565 7413 raft_consensus.cc:374] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.885589 7413 raft_consensus.cc:729] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.885632 7413 consensus_queue.cc:260] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.885710 7413 ts_tablet_manager.cc:1428] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:09.886858 7454 tablet_replica.cc:330] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: stopping tablet replica
I20260926 14:50:09.886976 7454 raft_consensus.cc:2229] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:09.887037 7454 raft_consensus.cc:2258] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:09.887116 7409 tablet_bootstrap.cc:492] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: No bootstrap required, opened a new log
I20260926 14:50:09.887166 7409 ts_tablet_manager.cc:1397] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 14:50:09.887290 7409 raft_consensus.cc:348] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.887365 7409 raft_consensus.cc:374] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:09.887391 7409 raft_consensus.cc:729] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Initialized, Role: FOLLOWER
I20260926 14:50:09.887419 7454 ts_tablet_manager.cc:1905] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:09.887431 7409 consensus_queue.cc:260] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.887512 7409 ts_tablet_manager.cc:1428] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:09.887719 7456 ts_tablet_manager.cc:1918] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:09.887779 7456 log.cc:1196] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/fdce2ffc4c534d379085ec3aa0ebbd4c
I20260926 14:50:09.888118 7456 ts_tablet_manager.cc:1939] T fdce2ffc4c534d379085ec3aa0ebbd4c P d11f5f75747e46329be1d9b83096610e: Deleting consensus metadata
I20260926 14:50:09.888718 6890 catalog_manager.cc:4841] TS d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545): tablet fdce2ffc4c534d379085ec3aa0ebbd4c (table test-table [id=fc2349b876ed4cc5b5e4a595cde8f875]) successfully deleted
I20260926 14:50:09.888779 7454 ts_tablet_manager.cc:1918] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:09.888835 7454 log.cc:1196] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/fdce2ffc4c534d379085ec3aa0ebbd4c
I20260926 14:50:09.889070 7454 ts_tablet_manager.cc:1939] T fdce2ffc4c534d379085ec3aa0ebbd4c P b3818094b7f74482b1afa7b92b736016: Deleting consensus metadata
I20260926 14:50:09.889606 6891 catalog_manager.cc:4841] TS b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623): tablet fdce2ffc4c534d379085ec3aa0ebbd4c (table test-table [id=fc2349b876ed4cc5b5e4a595cde8f875]) successfully deleted
I20260926 14:50:09.975279 7417 raft_consensus.cc:482] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:09.975390 7417 raft_consensus.cc:504] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.975713 7417 leader_election.cc:288] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805), d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:09.980283 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.976085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:46295 (local address 127.5.15.131:37545)
0926 14:50:09.976202 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.976204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.976520 (+ 316us) server_negotiation.cc:407] Connection header received
0926 14:50:09.976615 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.976616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.976625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.976652 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.976942 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.977697 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.979347 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.979909 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.979950 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.979958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.980177 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.980205 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 14:50:09.980347 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.976069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:33339 (local address 127.5.15.130:32805)
0926 14:50:09.976160 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:09.976163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:09.976291 (+ 128us) server_negotiation.cc:407] Connection header received
0926 14:50:09.976395 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:09.976397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:09.976407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:09.976436 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:09.976932 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.977706 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.979356 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.979916 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.979964 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.979973 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:09.980233 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 14:50:09.980266 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 14:50:09.980526 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.975905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37545 (local address 127.5.15.129:46295)
0926 14:50:09.976456 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.976469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.976546 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.976734 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.976735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.976738 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.976849 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.976851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.977838 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.977838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.979244 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.979247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.980008 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.980009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.980105 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.980109 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.980399 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.980438 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":133,"thread_start_us":77,"threads_started":1}
I20260926 14:50:09.980715 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9820027600d48d8901c7af440f88adb" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11f5f75747e46329be1d9b83096610e" is_pre_election: true
I20260926 14:50:09.980789 7343 raft_consensus.cc:2454] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 0.
I20260926 14:50:09.980937 6994 leader_election.cc:302] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3818094b7f74482b1afa7b92b736016, d11f5f75747e46329be1d9b83096610e; no voters:
I20260926 14:50:09.981048 7417 raft_consensus.cc:2781] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:09.981091 7417 raft_consensus.cc:482] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:09.981184 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:09.975897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:32805 (local address 127.5.15.129:33339)
0926 14:50:09.976222 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:09.976233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:09.976321 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:09.976738 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:09.976739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:09.976741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:09.976852 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.976853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.977807 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.977808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.979247 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:09.979251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:09.980058 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:09.980059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:09.980150 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:09.980155 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:09.980904 (+ 749us) client_negotiation.cc:240] Negotiation successful
0926 14:50:09.980950 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":25,"thread_start_us":72,"threads_started":1}
I20260926 14:50:09.981228 7417 raft_consensus.cc:3037] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.981699 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9820027600d48d8901c7af440f88adb" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b" is_pre_election: true
I20260926 14:50:09.981781 7203 raft_consensus.cc:2454] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 0.
I20260926 14:50:09.982383 7417 raft_consensus.cc:504] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.982507 7417 leader_election.cc:288] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 election: Requested vote from peers 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805), d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:09.982677 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9820027600d48d8901c7af440f88adb" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b"
I20260926 14:50:09.982770 7203 raft_consensus.cc:3037] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.982770 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9820027600d48d8901c7af440f88adb" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11f5f75747e46329be1d9b83096610e"
I20260926 14:50:09.982836 7343 raft_consensus.cc:3037] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:09.983508 7343 raft_consensus.cc:2454] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 1.
I20260926 14:50:09.983522 7203 raft_consensus.cc:2454] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 1.
I20260926 14:50:09.983691 6994 leader_election.cc:302] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3818094b7f74482b1afa7b92b736016, d11f5f75747e46329be1d9b83096610e; no voters:
I20260926 14:50:09.983798 7417 raft_consensus.cc:2781] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:09.983949 7417 raft_consensus.cc:686] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 1 LEADER]: Becoming Leader. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Running, Role: LEADER
I20260926 14:50:09.984012 7417 consensus_queue.cc:237] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:09.984779 6906 catalog_manager.cc:5495] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3818094b7f74482b1afa7b92b736016 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "b3818094b7f74482b1afa7b92b736016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:10.005630 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.003363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.5.15.129:44623)
0926 14:50:10.003405 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.003407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.003416 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:10.003476 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.003476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.003495 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.003561 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.003660 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.004160 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.004883 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.005087 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.005163 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.005306 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.005407 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.005485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.005527 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.005545 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:10.010211 7406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.008105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:58731 (local address 127.5.15.130:32805)
0926 14:50:10.008140 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.008142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.008300 (+ 158us) server_negotiation.cc:407] Connection header received
0926 14:50:10.008374 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.008375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.008387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.008428 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.008565 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.008989 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.009703 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.009871 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.009921 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.010048 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.010069 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.010100 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.010143 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.010156 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:10.010242 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.008046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:32805 (local address 127.5.15.129:58731)
0926 14:50:10.008228 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.008239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.008297 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.008442 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.008443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.008445 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:10.008510 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.008511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.009040 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.009041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.009649 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.009651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.009933 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.009933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.009979 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.009991 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.010113 (+ 122us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.010114 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.010143 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.010174 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260926 14:50:10.012131 7203 raft_consensus.cc:1261] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Refusing update from remote peer b3818094b7f74482b1afa7b92b736016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.012161 7343 raft_consensus.cc:1261] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Refusing update from remote peer b3818094b7f74482b1afa7b92b736016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:10.012382 7417 consensus_queue.cc:1035] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.012530 7462 consensus_queue.cc:1035] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:10.019735 7408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.017385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:47127 (local address 127.5.15.129:44623)
0926 14:50:10.017423 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:10.017424 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:10.017430 (+ 6us) server_negotiation.cc:407] Connection header received
0926 14:50:10.017447 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:10.017448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:10.017456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:10.017502 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:10.017623 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.018213 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.018937 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.019197 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.019267 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.019397 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.019427 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.019477 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:10.019631 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 14:50:10.019651 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:50:10.019749 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:10.017294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44623 (local address 127.5.15.130:47127)
0926 14:50:10.017329 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:10.017339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:10.017372 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:10.017512 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:10.017514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:10.017515 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:10.017554 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.017556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.018274 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.018275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.018873 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:10.018875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:10.019272 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:10.019273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:10.019332 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:10.019336 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:10.019546 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:10.019548 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:10.019626 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:50:10.019673 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 14:50:10.020088 7437 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:50:11.538677 6906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:11.538786 6906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:11.539542 6906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Sending DeleteTablet for 3 replicas of tablet f9820027600d48d8901c7af440f88adb
I20260926 14:50:11.539825 7183 tablet_service.cc:1514] Processing DeleteTablet for tablet f9820027600d48d8901c7af440f88adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:11 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:49504
I20260926 14:50:11.539815 7323 tablet_service.cc:1514] Processing DeleteTablet for tablet f9820027600d48d8901c7af440f88adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:11 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:52926
I20260926 14:50:11.539821 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet f9820027600d48d8901c7af440f88adb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:11 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:32768
I20260926 14:50:11.541695 7493 tablet_replica.cc:330] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: stopping tablet replica
I20260926 14:50:11.541772 7493 raft_consensus.cc:2229] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:11.541896 7494 tablet_replica.cc:330] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: stopping tablet replica
I20260926 14:50:11.541971 7494 raft_consensus.cc:2229] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:11.542098 7494 raft_consensus.cc:2258] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:11.542178 7493 raft_consensus.cc:2258] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:11.542428 7494 ts_tablet_manager.cc:1905] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:11.543850 7493 ts_tablet_manager.cc:1905] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:11.544030 7494 ts_tablet_manager.cc:1918] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:11.544081 7494 log.cc:1196] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/f9820027600d48d8901c7af440f88adb
I20260926 14:50:11.544327 7494 ts_tablet_manager.cc:1939] T f9820027600d48d8901c7af440f88adb P b3818094b7f74482b1afa7b92b736016: Deleting consensus metadata
I20260926 14:50:11.544649 6891 catalog_manager.cc:4841] TS b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623): tablet f9820027600d48d8901c7af440f88adb (table test-table [id=b3488bf0650e4986bbb93fe61cee202f]) successfully deleted
I20260926 14:50:11.545367 7493 ts_tablet_manager.cc:1918] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:11.545436 7493 log.cc:1196] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/f9820027600d48d8901c7af440f88adb
I20260926 14:50:11.545673 7493 ts_tablet_manager.cc:1939] T f9820027600d48d8901c7af440f88adb P 392289e8b0f044a99d089f74a8e8865b: Deleting consensus metadata
I20260926 14:50:11.545958 7492 tablet_replica.cc:330] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: stopping tablet replica
I20260926 14:50:11.545984 6893 catalog_manager.cc:4841] TS 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805): tablet f9820027600d48d8901c7af440f88adb (table test-table [id=b3488bf0650e4986bbb93fe61cee202f]) successfully deleted
I20260926 14:50:11.546025 7492 raft_consensus.cc:2229] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:11.546062 7492 raft_consensus.cc:2258] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:11.546270 7492 ts_tablet_manager.cc:1905] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:11.547627 7492 ts_tablet_manager.cc:1918] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:11.547703 7492 log.cc:1196] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/f9820027600d48d8901c7af440f88adb
I20260926 14:50:11.547904 7492 ts_tablet_manager.cc:1939] T f9820027600d48d8901c7af440f88adb P d11f5f75747e46329be1d9b83096610e: Deleting consensus metadata
I20260926 14:50:11.548197 6890 catalog_manager.cc:4841] TS d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545): tablet f9820027600d48d8901c7af440f88adb (table test-table [id=b3488bf0650e4986bbb93fe61cee202f]) successfully deleted
Sep 26 14:50:11 dist-test-slave-s4p1 krb5kdc[6861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434211, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:11 dist-test-slave-s4p1 krb5kdc[6861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434211, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:11.562868 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:11.555658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.5.15.190:42735)
0926 14:50:11.555796 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:11.555798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:11.555851 (+ 53us) server_negotiation.cc:407] Connection header received
0926 14:50:11.555930 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:11.555931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:11.555942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:11.556007 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:11.556573 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:11.557122 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:11.557854 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:11.558073 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:11.558152 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:11.560366 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:11.560367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:11.560367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:11.560374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:11.561824 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:11.562124 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:11.562125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:11.562125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:11.562179 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:11.562336 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:11.562336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:11.562337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:11.562455 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:11.562529 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:11.562613 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 14:50:11.562673 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260926 14:50:11.564196 6906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:11.564359 6906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:11.565646 7043 tablet_service.cc:1467] Processing CreateTablet for tablet e100a65d9e544042b939c00b613aa385 (DEFAULT_TABLE table=test-table [id=f3c37f31748c479f823b24058344b5ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:11.565657 7183 tablet_service.cc:1467] Processing CreateTablet for tablet e100a65d9e544042b939c00b613aa385 (DEFAULT_TABLE table=test-table [id=f3c37f31748c479f823b24058344b5ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:11.565725 7323 tablet_service.cc:1467] Processing CreateTablet for tablet e100a65d9e544042b939c00b613aa385 (DEFAULT_TABLE table=test-table [id=f3c37f31748c479f823b24058344b5ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:11.565779 7183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e100a65d9e544042b939c00b613aa385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:11.565783 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e100a65d9e544042b939c00b613aa385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:11.565846 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e100a65d9e544042b939c00b613aa385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:11.566879 7503 tablet_bootstrap.cc:492] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Bootstrap starting.
I20260926 14:50:11.567221 7504 tablet_bootstrap.cc:492] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Bootstrap starting.
I20260926 14:50:11.567291 7503 tablet_bootstrap.cc:654] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:11.567555 7504 tablet_bootstrap.cc:654] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:11.567847 7505 tablet_bootstrap.cc:492] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Bootstrap starting.
I20260926 14:50:11.567922 7503 tablet_bootstrap.cc:492] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: No bootstrap required, opened a new log
I20260926 14:50:11.568004 7503 ts_tablet_manager.cc:1397] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:11.568111 7504 tablet_bootstrap.cc:492] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: No bootstrap required, opened a new log
I20260926 14:50:11.568161 7504 ts_tablet_manager.cc:1397] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:11.568146 7503 raft_consensus.cc:348] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.568262 7503 raft_consensus.cc:374] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:11.568267 7505 tablet_bootstrap.cc:654] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:11.568316 7503 raft_consensus.cc:729] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Initialized, Role: FOLLOWER
I20260926 14:50:11.568320 7504 raft_consensus.cc:348] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.568408 7504 raft_consensus.cc:374] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:11.568440 7504 raft_consensus.cc:729] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392289e8b0f044a99d089f74a8e8865b, State: Initialized, Role: FOLLOWER
I20260926 14:50:11.568409 7503 consensus_queue.cc:260] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.568488 7504 consensus_queue.cc:260] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.568596 7503 ts_tablet_manager.cc:1428] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:11.568688 7504 ts_tablet_manager.cc:1428] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:11.569042 7505 tablet_bootstrap.cc:492] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: No bootstrap required, opened a new log
I20260926 14:50:11.569115 7505 ts_tablet_manager.cc:1397] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:11.569227 7505 raft_consensus.cc:348] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.569293 7505 raft_consensus.cc:374] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:11.569310 7505 raft_consensus.cc:729] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Initialized, Role: FOLLOWER
I20260926 14:50:11.569355 7505 consensus_queue.cc:260] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.569438 7505 ts_tablet_manager.cc:1428] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:11.591789 7483 raft_consensus.cc:482] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:11.591900 7483 raft_consensus.cc:504] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.592090 7483 leader_election.cc:288] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805), d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:11.592285 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e100a65d9e544042b939c00b613aa385" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b" is_pre_election: true
I20260926 14:50:11.592278 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e100a65d9e544042b939c00b613aa385" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11f5f75747e46329be1d9b83096610e" is_pre_election: true
I20260926 14:50:11.592379 7343 raft_consensus.cc:2454] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 0.
I20260926 14:50:11.592379 7203 raft_consensus.cc:2454] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 0.
I20260926 14:50:11.592532 6997 leader_election.cc:302] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392289e8b0f044a99d089f74a8e8865b, b3818094b7f74482b1afa7b92b736016; no voters:
I20260926 14:50:11.592623 7483 raft_consensus.cc:2781] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:11.592674 7483 raft_consensus.cc:482] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:11.592706 7483 raft_consensus.cc:3037] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:11.593343 7483 raft_consensus.cc:504] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.593472 7483 leader_election.cc:288] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 election: Requested vote from peers 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805), d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545)
I20260926 14:50:11.593632 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e100a65d9e544042b939c00b613aa385" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b"
I20260926 14:50:11.593710 7203 raft_consensus.cc:3037] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:11.593719 7343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e100a65d9e544042b939c00b613aa385" candidate_uuid: "b3818094b7f74482b1afa7b92b736016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11f5f75747e46329be1d9b83096610e"
I20260926 14:50:11.593803 7343 raft_consensus.cc:3037] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:11.594308 7203 raft_consensus.cc:2454] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 1.
I20260926 14:50:11.594467 7343 raft_consensus.cc:2454] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3818094b7f74482b1afa7b92b736016 in term 1.
I20260926 14:50:11.594529 6997 leader_election.cc:302] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 392289e8b0f044a99d089f74a8e8865b, b3818094b7f74482b1afa7b92b736016; no voters:
I20260926 14:50:11.594643 7483 raft_consensus.cc:2781] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:11.594699 7483 raft_consensus.cc:686] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 1 LEADER]: Becoming Leader. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Running, Role: LEADER
I20260926 14:50:11.594771 7483 consensus_queue.cc:237] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.595350 6905 catalog_manager.cc:5495] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3818094b7f74482b1afa7b92b736016 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "b3818094b7f74482b1afa7b92b736016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:11.607786 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:11.605626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.5.15.129:44623)
0926 14:50:11.605798 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:11.605800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:11.605807 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:11.605825 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:11.605826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:11.605835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:11.605892 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:11.605999 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:11.606481 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:11.607199 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:11.607384 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:11.607434 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:11.607535 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:11.607559 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:11.607593 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:11.607661 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 14:50:11.607676 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260926 14:50:11.609740 7343 raft_consensus.cc:1261] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Refusing update from remote peer b3818094b7f74482b1afa7b92b736016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:11.609735 7203 raft_consensus.cc:1261] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Refusing update from remote peer b3818094b7f74482b1afa7b92b736016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:11.609958 7483 consensus_queue.cc:1035] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:11.610070 7462 consensus_queue.cc:1035] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:11.619267 6906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:11.619341 6906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:11.619975 6906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Sending DeleteTablet for 3 replicas of tablet e100a65d9e544042b939c00b613aa385
I20260926 14:50:11.620183 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet e100a65d9e544042b939c00b613aa385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:11 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:32768
I20260926 14:50:11.620200 7323 tablet_service.cc:1514] Processing DeleteTablet for tablet e100a65d9e544042b939c00b613aa385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:11 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:52926
I20260926 14:50:11.620285 7492 tablet_replica.cc:330] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: stopping tablet replica
I20260926 14:50:11.620280 7494 tablet_replica.cc:330] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: stopping tablet replica
I20260926 14:50:11.620330 7494 raft_consensus.cc:2229] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:11.620333 7492 raft_consensus.cc:2229] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:11.620366 7492 raft_consensus.cc:2258] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:11.620379 7494 raft_consensus.cc:2258] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:11.620469 6906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:50:11.620525 7183 tablet_service.cc:1514] Processing DeleteTablet for tablet e100a65d9e544042b939c00b613aa385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:11 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:49504
W20260926 14:50:11.620594 6906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:11.620599 7493 tablet_replica.cc:330] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: stopping tablet replica
I20260926 14:50:11.620644 7494 ts_tablet_manager.cc:1905] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:11.620659 7493 raft_consensus.cc:2229] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:11.620697 7493 raft_consensus.cc:2258] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:11.620769 7492 ts_tablet_manager.cc:1905] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:11.620968 7493 ts_tablet_manager.cc:1905] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:11.622251 7492 ts_tablet_manager.cc:1918] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:11.622332 7492 log.cc:1196] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/e100a65d9e544042b939c00b613aa385
I20260926 14:50:11.622471 7493 ts_tablet_manager.cc:1918] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:11.622520 7493 log.cc:1196] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/e100a65d9e544042b939c00b613aa385
I20260926 14:50:11.622521 7494 ts_tablet_manager.cc:1918] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:11.622565 7494 log.cc:1196] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/e100a65d9e544042b939c00b613aa385
I20260926 14:50:11.622565 7492 ts_tablet_manager.cc:1939] T e100a65d9e544042b939c00b613aa385 P d11f5f75747e46329be1d9b83096610e: Deleting consensus metadata
I20260926 14:50:11.622777 7493 ts_tablet_manager.cc:1939] T e100a65d9e544042b939c00b613aa385 P 392289e8b0f044a99d089f74a8e8865b: Deleting consensus metadata
I20260926 14:50:11.622795 7494 ts_tablet_manager.cc:1939] T e100a65d9e544042b939c00b613aa385 P b3818094b7f74482b1afa7b92b736016: Deleting consensus metadata
I20260926 14:50:11.622937 6890 catalog_manager.cc:4841] TS d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545): tablet e100a65d9e544042b939c00b613aa385 (table test-table [id=f3c37f31748c479f823b24058344b5ac]) successfully deleted
I20260926 14:50:11.623082 6891 catalog_manager.cc:4841] TS b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623): tablet e100a65d9e544042b939c00b613aa385 (table test-table [id=f3c37f31748c479f823b24058344b5ac]) successfully deleted
I20260926 14:50:11.623283 6893 catalog_manager.cc:4841] TS 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805): tablet e100a65d9e544042b939c00b613aa385 (table test-table [id=f3c37f31748c479f823b24058344b5ac]) successfully deleted
I20260926 14:50:11.623332 7183 tablet_service.cc:1467] Processing CreateTablet for tablet c2a88444bd334294b494fa64e8a3c88d (DEFAULT_TABLE table=test-table [id=46de1fce86a547108717812f63844642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:11.623339 7323 tablet_service.cc:1467] Processing CreateTablet for tablet c2a88444bd334294b494fa64e8a3c88d (DEFAULT_TABLE table=test-table [id=46de1fce86a547108717812f63844642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:11.623432 7323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a88444bd334294b494fa64e8a3c88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:11.623432 7183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a88444bd334294b494fa64e8a3c88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:11.624221 7503 tablet_bootstrap.cc:492] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Bootstrap starting.
I20260926 14:50:11.624224 7504 tablet_bootstrap.cc:492] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Bootstrap starting.
I20260926 14:50:11.624348 7043 tablet_service.cc:1467] Processing CreateTablet for tablet c2a88444bd334294b494fa64e8a3c88d (DEFAULT_TABLE table=test-table [id=46de1fce86a547108717812f63844642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:11.624436 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a88444bd334294b494fa64e8a3c88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:11.624521 7503 tablet_bootstrap.cc:654] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:11.624584 7504 tablet_bootstrap.cc:654] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:11.625079 7503 tablet_bootstrap.cc:492] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: No bootstrap required, opened a new log
I20260926 14:50:11.625123 7504 tablet_bootstrap.cc:492] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: No bootstrap required, opened a new log
I20260926 14:50:11.625137 7503 ts_tablet_manager.cc:1397] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:11.625159 7504 ts_tablet_manager.cc:1397] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:11.625257 7504 raft_consensus.cc:348] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.625267 7503 raft_consensus.cc:348] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.625319 7504 raft_consensus.cc:374] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:11.625321 7503 raft_consensus.cc:374] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:11.625344 7504 raft_consensus.cc:729] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 392289e8b0f044a99d089f74a8e8865b, State: Initialized, Role: FOLLOWER
I20260926 14:50:11.625345 7503 raft_consensus.cc:729] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Initialized, Role: FOLLOWER
I20260926 14:50:11.625386 7504 consensus_queue.cc:260] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.625391 7503 consensus_queue.cc:260] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.625475 7504 ts_tablet_manager.cc:1428] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:11.625475 7503 ts_tablet_manager.cc:1428] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:11.625626 7505 tablet_bootstrap.cc:492] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016: Bootstrap starting.
I20260926 14:50:11.625921 7505 tablet_bootstrap.cc:654] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:11.626570 7505 tablet_bootstrap.cc:492] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016: No bootstrap required, opened a new log
I20260926 14:50:11.626621 7505 ts_tablet_manager.cc:1397] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:11.626732 7505 raft_consensus.cc:348] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.626817 7505 raft_consensus.cc:374] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:11.626838 7505 raft_consensus.cc:729] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3818094b7f74482b1afa7b92b736016, State: Initialized, Role: FOLLOWER
I20260926 14:50:11.626888 7505 consensus_queue.cc:260] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.626976 7505 ts_tablet_manager.cc:1428] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:11.706925 7509 raft_consensus.cc:482] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:11.707021 7509 raft_consensus.cc:504] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.707193 7509 leader_election.cc:288] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623), 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805)
I20260926 14:50:11.707399 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a88444bd334294b494fa64e8a3c88d" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3818094b7f74482b1afa7b92b736016" is_pre_election: true
I20260926 14:50:11.707418 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a88444bd334294b494fa64e8a3c88d" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b" is_pre_election: true
I20260926 14:50:11.707497 7063 raft_consensus.cc:2454] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 0.
I20260926 14:50:11.707507 7203 raft_consensus.cc:2454] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 0.
I20260926 14:50:11.707659 7275 leader_election.cc:302] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3818094b7f74482b1afa7b92b736016, d11f5f75747e46329be1d9b83096610e; no voters:
I20260926 14:50:11.707746 7509 raft_consensus.cc:2781] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:11.707811 7509 raft_consensus.cc:482] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:11.707836 7509 raft_consensus.cc:3037] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:11.708568 7509 raft_consensus.cc:504] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.708720 7509 leader_election.cc:288] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 election: Requested vote from peers b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623), 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805)
I20260926 14:50:11.708864 7063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a88444bd334294b494fa64e8a3c88d" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3818094b7f74482b1afa7b92b736016"
I20260926 14:50:11.708878 7203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a88444bd334294b494fa64e8a3c88d" candidate_uuid: "d11f5f75747e46329be1d9b83096610e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "392289e8b0f044a99d089f74a8e8865b"
I20260926 14:50:11.708945 7063 raft_consensus.cc:3037] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:11.708983 7203 raft_consensus.cc:3037] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:11.709573 7063 raft_consensus.cc:2454] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 1.
I20260926 14:50:11.709651 7203 raft_consensus.cc:2454] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11f5f75747e46329be1d9b83096610e in term 1.
I20260926 14:50:11.709748 7275 leader_election.cc:302] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3818094b7f74482b1afa7b92b736016, d11f5f75747e46329be1d9b83096610e; no voters:
I20260926 14:50:11.709872 7509 raft_consensus.cc:2781] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:11.709966 7509 raft_consensus.cc:686] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 1 LEADER]: Becoming Leader. State: Replica: d11f5f75747e46329be1d9b83096610e, State: Running, Role: LEADER
I20260926 14:50:11.710006 7509 consensus_queue.cc:237] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } }
I20260926 14:50:11.710601 6906 catalog_manager.cc:5495] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e reported cstate change: term changed from 0 to 1, leader changed from <none> to d11f5f75747e46329be1d9b83096610e (127.5.15.131). New cstate: current_term: 1 leader_uuid: "d11f5f75747e46329be1d9b83096610e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11f5f75747e46329be1d9b83096610e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37545 } health_report { overall_health: HEALTHY } } }
I20260926 14:50:11.721311 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:11.719331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.5.15.131:37545)
0926 14:50:11.719476 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:11.719478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:11.719486 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:11.719510 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:11.719524 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:11.719534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:11.719592 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:11.719691 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:11.720151 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:11.720770 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:11.720942 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:11.720986 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:11.721097 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:11.721129 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:11.721165 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:11.721202 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 14:50:11.721217 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":102,"threads_started":1}
I20260926 14:50:11.723253 7203 raft_consensus.cc:1261] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Refusing update from remote peer d11f5f75747e46329be1d9b83096610e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:11.723265 7063 raft_consensus.cc:1261] T c2a88444bd334294b494fa64e8a3c88d P b3818094b7f74482b1afa7b92b736016 [term 1 FOLLOWER]: Refusing update from remote peer d11f5f75747e46329be1d9b83096610e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:11.723438 7509 consensus_queue.cc:1035] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "392289e8b0f044a99d089f74a8e8865b" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:11.723517 7520 consensus_queue.cc:1035] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3818094b7f74482b1afa7b92b736016" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:50:11.727928 7476 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:50:13.243124 6905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:13.243234 6905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:13.244060 6905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00efa4cafa1c46989ae4a20ef9ae66f4: Sending DeleteTablet for 3 replicas of tablet c2a88444bd334294b494fa64e8a3c88d
I20260926 14:50:13.244304 7043 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a88444bd334294b494fa64e8a3c88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:13 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:32768
I20260926 14:50:13.244308 7183 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a88444bd334294b494fa64e8a3c88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:13 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:49504
I20260926 14:50:13.244537 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6978
I20260926 14:50:13.244539 7323 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a88444bd334294b494fa64e8a3c88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:13 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:52926
I20260926 14:50:13.245930 7549 tablet_replica.cc:330] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: stopping tablet replica
I20260926 14:50:13.246002 7549 raft_consensus.cc:2229] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:13.246100 7549 raft_consensus.cc:2258] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:13.246382 7549 ts_tablet_manager.cc:1905] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:13.246838 7547 tablet_replica.cc:330] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: stopping tablet replica
I20260926 14:50:13.246898 7547 raft_consensus.cc:2229] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:13.246964 7547 raft_consensus.cc:2258] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:13.247191 7547 ts_tablet_manager.cc:1905] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:13.248131 7549 ts_tablet_manager.cc:1918] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:13.248194 7549 log.cc:1196] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/c2a88444bd334294b494fa64e8a3c88d
I20260926 14:50:13.248409 7549 ts_tablet_manager.cc:1939] T c2a88444bd334294b494fa64e8a3c88d P d11f5f75747e46329be1d9b83096610e: Deleting consensus metadata
I20260926 14:50:13.248742 6890 catalog_manager.cc:4841] TS d11f5f75747e46329be1d9b83096610e (127.5.15.131:37545): tablet c2a88444bd334294b494fa64e8a3c88d (table test-table [id=46de1fce86a547108717812f63844642]) successfully deleted
I20260926 14:50:13.248914 7547 ts_tablet_manager.cc:1918] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:50:13.248968 7547 log.cc:1196] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/c2a88444bd334294b494fa64e8a3c88d
I20260926 14:50:13.249187 7547 ts_tablet_manager.cc:1939] T c2a88444bd334294b494fa64e8a3c88d P 392289e8b0f044a99d089f74a8e8865b: Deleting consensus metadata
I20260926 14:50:13.249496 6893 catalog_manager.cc:4841] TS 392289e8b0f044a99d089f74a8e8865b (127.5.15.130:32805): tablet c2a88444bd334294b494fa64e8a3c88d (table test-table [id=46de1fce86a547108717812f63844642]) successfully deleted
W20260926 14:50:13.251454 6891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:44623 (error 108)
I20260926 14:50:13.251701 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 7118
I20260926 14:50:13.252863 7551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:13.252679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44623 (local address 127.0.0.1:56076)
0926 14:50:13.252781 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:13.252828 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:44623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":62,"threads_started":1}
W20260926 14:50:13.253005 6891 catalog_manager.cc:4584] TS b3818094b7f74482b1afa7b92b736016 (127.5.15.129:44623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c2a88444bd334294b494fa64e8a3c88d: Network error: Client connection negotiation failed: client connection to 127.5.15.129:44623: connect: Connection refused (error 111)
I20260926 14:50:13.257782 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 7258
I20260926 14:50:13.262938 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 6874
2026-09-26T14:50:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5426 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:13 dist-test-slave-s4p1 krb5kdc[7554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:13 dist-test-slave-s4p1 krb5kdc[7554](info): set up 2 sockets
Sep 26 14:50:13 dist-test-slave-s4p1 krb5kdc[7554](info): commencing operation
krb5kdc: starting...
W20260926 14:50:14.290493 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:14.406814 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:39627
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:14.485426 7567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:14.485674 7567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:14.485711 7567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:14.486920 7567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:14.486990 7567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:14.487013 7567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:14.487032 7567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:14.487090 7567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:14.488647 7567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:39627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:14.488940 7567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:14.489179 7567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:14.493868 7573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:14.493908 7575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:14.493990 7572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:14.494185 7567 server_base.cc:1034] running on GCE node
I20260926 14:50:14.494410 7567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:14.494654 7567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:14.495792 7567 hybrid_clock.cc:648] HybridClock initialized: now 1790434214495784 us; error 38 us; skew 500 ppm
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:14.498224 7567 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:14.498662 7567 webserver.cc:460] Webserver started at http://127.5.15.190:40321/ using document root <none> and password file <none>
I20260926 14:50:14.498920 7567 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:14.498989 7567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:14.499143 7567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:14.500200 7567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "f0422acf1af942c7845ea5c13148ff91"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "01b1b5e79bf82926677928010ead3185"
server_key_iv: "5910fec890e69b5c2f11caef7fd5ce1c"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.500564 7567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "f0422acf1af942c7845ea5c13148ff91"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "01b1b5e79bf82926677928010ead3185"
server_key_iv: "5910fec890e69b5c2f11caef7fd5ce1c"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.501844 7567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:14.502517 7582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.502682 7567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:14.502789 7567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "f0422acf1af942c7845ea5c13148ff91"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "01b1b5e79bf82926677928010ead3185"
server_key_iv: "5910fec890e69b5c2f11caef7fd5ce1c"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.502871 7567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:14.523619 7567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:14.525960 7567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:14.526129 7567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:14.530047 7567 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:39627
I20260926 14:50:14.530094 7634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:39627 every 8 connection(s)
I20260926 14:50:14.530445 7567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:14.530912 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 7567
I20260926 14:50:14.531011 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:14.531199 5182 external_mini_cluster.cc:1428] Setting key 2b9b9fcdb1d2030c4d53022b24871baf
I20260926 14:50:14.531530 7635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:14.534548 7635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91: Bootstrap starting.
I20260926 14:50:14.535295 7635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:14.535573 7635 log.cc:824] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:14.536347 7635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91: No bootstrap required, opened a new log
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:14.538259 7635 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0422acf1af942c7845ea5c13148ff91" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39627 } }
I20260926 14:50:14.538446 7635 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:14.538492 7635 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0422acf1af942c7845ea5c13148ff91, State: Initialized, Role: FOLLOWER
I20260926 14:50:14.538674 7635 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0422acf1af942c7845ea5c13148ff91" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39627 } }
I20260926 14:50:14.538784 7635 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:14.538827 7635 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:14.538884 7635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:14.539605 7635 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0422acf1af942c7845ea5c13148ff91" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39627 } }
I20260926 14:50:14.539723 7635 leader_election.cc:302] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0422acf1af942c7845ea5c13148ff91; no voters:
I20260926 14:50:14.539883 7635 leader_election.cc:288] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:14.539949 7640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:14.540122 7635 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:14.540130 7640 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [term 1 LEADER]: Becoming Leader. State: Replica: f0422acf1af942c7845ea5c13148ff91, State: Running, Role: LEADER
I20260926 14:50:14.540314 7640 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0422acf1af942c7845ea5c13148ff91" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39627 } }
I20260926 14:50:14.540664 7641 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0422acf1af942c7845ea5c13148ff91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0422acf1af942c7845ea5c13148ff91" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39627 } } }
I20260926 14:50:14.540736 7641 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:14.540882 7642 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0422acf1af942c7845ea5c13148ff91. Latest consensus state: current_term: 1 leader_uuid: "f0422acf1af942c7845ea5c13148ff91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0422acf1af942c7845ea5c13148ff91" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39627 } } }
I20260926 14:50:14.540937 7642 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:14.541358 7648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:14.541920 7648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:14.542411 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:14.531702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.5.15.190:39627)
0926 14:50:14.531872 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:14.531880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:14.531913 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:14.532577 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:14.532584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:14.532630 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:14.532740 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:14.533474 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.534305 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.535385 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.535860 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.535940 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:14.538190 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:14.538194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:14.538194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:14.538206 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:14.540273 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:14.540693 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:14.540697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:14.540697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:14.540813 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:14.541048 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:14.541049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:14.541049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:14.541375 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:14.542005 (+ 630us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:14.542088 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:50:14.542229 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260926 14:50:14.543440 7648 catalog_manager.cc:1347] Generated new cluster ID: 76be08a7e8024704929027dc3957bd05
I20260926 14:50:14.543480 7648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:14.555325 7648 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:14.555838 7648 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:14.573385 7648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0422acf1af942c7845ea5c13148ff91: Generated new TSK 0
I20260926 14:50:14.573611 7648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:14.607401 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:39627
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:14.704342 7661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:14.704525 7661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:14.704547 7661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:14.705672 7661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:14.705713 7661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:14.705739 7661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:14.707221 7661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:14.707399 7661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:14.707572 7661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:14.712314 7667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:14.712312 7666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:14.712399 7661 server_base.cc:1034] running on GCE node
W20260926 14:50:14.712317 7669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:14.712721 7661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:14.712975 7661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:14.714143 7661 hybrid_clock.cc:648] HybridClock initialized: now 1790434214714137 us; error 34 us; skew 500 ppm
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:14.716544 7661 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:14.716984 7661 webserver.cc:460] Webserver started at http://127.5.15.129:41451/ using document root <none> and password file <none>
I20260926 14:50:14.717216 7661 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:14.717284 7661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:14.717432 7661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:14.718439 7661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "a6e3d63ccb734d53bbfd3eae4e13df03"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "57c9f8466e0f05e6fda3c1a7e6c6d9e8"
server_key_iv: "a41716d370fa5f47a293ba86019a4a43"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.718864 7661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "a6e3d63ccb734d53bbfd3eae4e13df03"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "57c9f8466e0f05e6fda3c1a7e6c6d9e8"
server_key_iv: "a41716d370fa5f47a293ba86019a4a43"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.720149 7661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:14.720844 7676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.721004 7661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:14.721103 7661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "a6e3d63ccb734d53bbfd3eae4e13df03"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "57c9f8466e0f05e6fda3c1a7e6c6d9e8"
server_key_iv: "a41716d370fa5f47a293ba86019a4a43"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.721186 7661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:14.733979 7661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:14.735823 7661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:14.735977 7661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:14.736203 7661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:14.736493 7661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:14.736574 7661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.736629 7661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:14.736675 7661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.742841 7661 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:44841
I20260926 14:50:14.742888 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:44841 every 8 connection(s)
I20260926 14:50:14.743204 7661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:14.752367 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:14.743908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:50963 (local address 127.5.15.190:39627)
0926 14:50:14.743968 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:14.743970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:14.744185 (+ 215us) server_negotiation.cc:407] Connection header received
0926 14:50:14.744682 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:14.744683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:14.744694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:14.744744 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:14.745698 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.746201 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.747298 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.747533 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.747620 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:14.749710 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:14.749711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:14.749712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:14.749719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:14.751011 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:14.751407 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:14.751408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:14.751409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:14.751469 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:14.751710 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:14.751711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:14.751711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:14.751829 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:14.751903 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:14.752187 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 14:50:14.752277 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:50:14.752656 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 7661
I20260926 14:50:14.752758 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:14.752934 5182 external_mini_cluster.cc:1428] Setting key 7de3d26c44252fccd789eb8dccecf3c2
I20260926 14:50:14.753063 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:14.743897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:39627 (local address 127.5.15.129:50963)
0926 14:50:14.744064 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:14.744103 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:14.744580 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:14.744810 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:14.744814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:14.745394 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:14.745580 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:14.745585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.746298 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.746299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:14.747205 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:14.747208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.747627 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.747629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:14.747751 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:14.748292 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:14.748300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:14.749614 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:14.751099 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:14.751106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:14.751108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:14.751319 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:14.751535 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:14.751536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:14.751536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:14.751640 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:14.751907 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:14.751913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:14.752049 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 14:50:14.752152 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:50:14.752304 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260926 14:50:14.753615 7790 heartbeater.cc:347] Connected to a master server at 127.5.15.190:39627
I20260926 14:50:14.753718 7790 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:14.753914 7790 heartbeater.cc:511] Master 127.5.15.190:39627 requested a full tablet report, sending...
I20260926 14:50:14.754415 7599 ts_manager.cc:194] Registered new tserver with Master: a6e3d63ccb734d53bbfd3eae4e13df03 (127.5.15.129:44841)
I20260926 14:50:14.755112 7599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:50963
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:14.775833 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:39627
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:14.853855 7795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:14.854036 7795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:14.854058 7795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:14.855167 7795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:14.855203 7795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:14.855229 7795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:14.856535 7795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:14.856700 7795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:14.856849 7795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:14.861413 7803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:14.861425 7801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:14.861420 7800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:14.861680 7795 server_base.cc:1034] running on GCE node
I20260926 14:50:14.861882 7795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:14.862110 7795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:14.863263 7795 hybrid_clock.cc:648] HybridClock initialized: now 1790434214863250 us; error 26 us; skew 500 ppm
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:14.865491 7795 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:14.865924 7795 webserver.cc:460] Webserver started at http://127.5.15.130:44873/ using document root <none> and password file <none>
I20260926 14:50:14.866142 7795 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:14.866206 7795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:14.866364 7795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:14.867370 7795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "621c80f09eb34f1b971206f4170f87c6"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "ccb492d48bf9afe05c13cada888c1462"
server_key_iv: "c27e6d25d9c960f301d5aa8fce0abcad"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.867707 7795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "621c80f09eb34f1b971206f4170f87c6"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "ccb492d48bf9afe05c13cada888c1462"
server_key_iv: "c27e6d25d9c960f301d5aa8fce0abcad"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.868880 7795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:14.869545 7810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.869714 7795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:14.869804 7795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "621c80f09eb34f1b971206f4170f87c6"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "ccb492d48bf9afe05c13cada888c1462"
server_key_iv: "c27e6d25d9c960f301d5aa8fce0abcad"
server_key_version: "encryptionkey@0"
I20260926 14:50:14.869885 7795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:14.878443 7795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:14.880223 7795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:14.880370 7795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:14.880579 7795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:14.880862 7795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:14.880895 7795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.880914 7795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:14.880961 7795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:14.886348 7795 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:38847
I20260926 14:50:14.886431 7923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:38847 every 8 connection(s)
I20260926 14:50:14.886688 7795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:14.890084 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 7795
I20260926 14:50:14.890177 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:14.890362 5182 external_mini_cluster.cc:1428] Setting key e69eb8fea1d385ca7639e0f0a2a63e48
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:14.896708 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:14.887378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:45481 (local address 127.5.15.190:39627)
0926 14:50:14.887460 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:14.887462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:14.887634 (+ 172us) server_negotiation.cc:407] Connection header received
0926 14:50:14.888078 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:14.888079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:14.888088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:14.888140 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:14.889075 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.889559 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.890941 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.891257 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.891310 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:14.893860 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:14.893862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:14.893862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:14.893868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:14.895296 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:14.895783 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:14.895783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:14.895784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:14.895831 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:14.896121 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:14.896122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:14.896122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:14.896215 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:14.896259 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:14.896593 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 14:50:14.896650 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:14.896972 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:14.887324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:39627 (local address 127.5.15.130:45481)
0926 14:50:14.887530 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:14.887569 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:14.887993 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:14.888188 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:14.888191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:14.888715 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:14.888990 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:14.888995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.889634 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.889634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:14.890866 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:14.890868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:14.891340 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:14.891341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:14.891478 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:14.891949 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:14.891958 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:14.893772 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:14.895378 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:14.895385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:14.895387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:14.895698 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:14.895894 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:14.895895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:14.895896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:14.896048 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:14.896277 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:14.896283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:14.896458 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 14:50:14.896578 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 14:50:14.896747 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":139,"threads_started":1}
I20260926 14:50:14.897456 7924 heartbeater.cc:347] Connected to a master server at 127.5.15.190:39627
I20260926 14:50:14.897559 7924 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:14.897737 7924 heartbeater.cc:511] Master 127.5.15.190:39627 requested a full tablet report, sending...
I20260926 14:50:14.898094 7599 ts_manager.cc:194] Registered new tserver with Master: 621c80f09eb34f1b971206f4170f87c6 (127.5.15.130:38847)
I20260926 14:50:14.898478 7599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:45481
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:14.915275 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:39627
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:14.992697 7929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:14.992882 7929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:14.992903 7929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:14.994153 7929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:14.994194 7929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:14.994220 7929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:14.995641 7929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:14.995877 7929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:14.996102 7929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:15.000818 7935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:15.000869 7934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:15.000831 7937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:15.001166 7929 server_base.cc:1034] running on GCE node
I20260926 14:50:15.001348 7929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:15.001581 7929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:15.002785 7929 hybrid_clock.cc:648] HybridClock initialized: now 1790434215002773 us; error 29 us; skew 500 ppm
Sep 26 14:50:14 dist-test-slave-s4p1 krb5kdc[7554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:15.005185 7929 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:15.005596 7929 webserver.cc:460] Webserver started at http://127.5.15.131:37467/ using document root <none> and password file <none>
I20260926 14:50:15.005787 7929 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:15.005827 7929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:15.005913 7929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:15.007074 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "4c853091e1c04151a94be328be5cb158"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "d37b3cdf10f626aa0d30ba89f2d5a27f"
server_key_iv: "8e8aa07e292668154f9ecea56e31f4f3"
server_key_version: "encryptionkey@0"
I20260926 14:50:15.007376 7929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "4c853091e1c04151a94be328be5cb158"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "d37b3cdf10f626aa0d30ba89f2d5a27f"
server_key_iv: "8e8aa07e292668154f9ecea56e31f4f3"
server_key_version: "encryptionkey@0"
I20260926 14:50:15.008526 7929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:15.009191 7944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:15.009361 7929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:15.009443 7929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "4c853091e1c04151a94be328be5cb158"
format_stamp: "Formatted at 2026-09-26 14:50:14 on dist-test-slave-s4p1"
server_key: "d37b3cdf10f626aa0d30ba89f2d5a27f"
server_key_iv: "8e8aa07e292668154f9ecea56e31f4f3"
server_key_version: "encryptionkey@0"
I20260926 14:50:15.009533 7929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:15.043793 7929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:15.045674 7929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:15.045821 7929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:15.046077 7929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:15.046403 7929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:15.046437 7929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:15.046478 7929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:15.046502 7929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:15.052104 7929 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:45647
I20260926 14:50:15.052145 8057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:45647 every 8 connection(s)
I20260926 14:50:15.052465 7929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:15.059587 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 7929
Sep 26 14:50:15 dist-test-slave-s4p1 krb5kdc[7554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:15.059692 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:15.059854 5182 external_mini_cluster.cc:1428] Setting key f95116f53adc0c80271a90a3d8ff8855
I20260926 14:50:15.062969 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.053165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:37971 (local address 127.5.15.190:39627)
0926 14:50:15.053225 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:15.053226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:15.053430 (+ 204us) server_negotiation.cc:407] Connection header received
0926 14:50:15.053966 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:15.053967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:15.053980 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:15.054042 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:15.055244 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.056088 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.057666 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.057877 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.057961 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.060157 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:15.060158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:15.060159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:15.060166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:15.061579 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:15.061969 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:15.061970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:15.061971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:15.062032 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:15.062302 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:15.062303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:15.062303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:15.062414 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:15.062491 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:15.062785 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 14:50:15.062884 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:50:15.063160 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:15.053149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:39627 (local address 127.5.15.131:37971)
0926 14:50:15.053317 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:15.053352 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:15.053854 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:15.054109 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:15.054113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:15.054834 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:15.055093 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:15.055099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.056182 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.056184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:15.057563 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:15.057566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:15.057970 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:15.057972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:15.058122 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:15.058564 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:15.058572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:15.059978 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:15.061659 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:15.061666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:15.061668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:15.061880 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:15.062100 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:15.062100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:15.062101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:15.062222 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:15.062480 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:15.062486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:15.062639 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 14:50:15.062747 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:50:15.062924 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260926 14:50:15.063555 8058 heartbeater.cc:347] Connected to a master server at 127.5.15.190:39627
I20260926 14:50:15.063658 8058 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:15.063824 8058 heartbeater.cc:511] Master 127.5.15.190:39627 requested a full tablet report, sending...
I20260926 14:50:15.064172 7599 ts_manager.cc:194] Registered new tserver with Master: 4c853091e1c04151a94be328be5cb158 (127.5.15.131:45647)
I20260926 14:50:15.064502 7599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:37971
I20260926 14:50:15.071542 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:50:15.082232 7638 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:15.081389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.5.15.190:39627)
0926 14:50:15.081450 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:15.081457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:15.081564 (+ 107us) server_negotiation.cc:407] Connection header received
0926 14:50:15.081606 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:15.081607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:15.081616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:15.081670 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:15.082145 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47942: BlockingRecv error: recv got EOF from 127.0.0.1:47942 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:50:15.083168 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 7661
I20260926 14:50:15.087595 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 7795
I20260926 14:50:15.091796 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 7929
I20260926 14:50:15.095548 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 7567
2026-09-26T14:50:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1831 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:15 dist-test-slave-s4p1 krb5kdc[8070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:15 dist-test-slave-s4p1 krb5kdc[8070](info): set up 2 sockets
Sep 26 14:50:15 dist-test-slave-s4p1 krb5kdc[8070](info): commencing operation
krb5kdc: starting...
W20260926 14:50:16.102448 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:16 dist-test-slave-s4p1 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:16.213526 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44829
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:44829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:16.290364 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:16.290542 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:16.290563 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:16.291604 8083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:16.291643 8083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:16.291656 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:16.291669 8083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:16.291680 8083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:16.293208 8083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:44829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:16.293386 8083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:16.293540 8083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:16.298202 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:16.298321 8083 server_base.cc:1034] running on GCE node
W20260926 14:50:16.298210 8088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:16.298190 8091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:16.298796 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:16.299086 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:16.300231 8083 hybrid_clock.cc:648] HybridClock initialized: now 1790434216300220 us; error 28 us; skew 500 ppm
Sep 26 14:50:16 dist-test-slave-s4p1 krb5kdc[8070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434216, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:16.302536 8083 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:16.303012 8083 webserver.cc:460] Webserver started at http://127.5.15.190:41977/ using document root <none> and password file <none>
I20260926 14:50:16.303220 8083 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:16.303261 8083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:16.303416 8083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:16.304390 8083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "cd8f91be279347078b2db83d30caa870"
format_stamp: "Formatted at 2026-09-26 14:50:16 on dist-test-slave-s4p1"
server_key: "96b09e31c1ad46b394a37ec0865113e9"
server_key_iv: "cb66ba67c0b00234bf3aa05bd3f11dc0"
server_key_version: "encryptionkey@0"
I20260926 14:50:16.304733 8083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "cd8f91be279347078b2db83d30caa870"
format_stamp: "Formatted at 2026-09-26 14:50:16 on dist-test-slave-s4p1"
server_key: "96b09e31c1ad46b394a37ec0865113e9"
server_key_iv: "cb66ba67c0b00234bf3aa05bd3f11dc0"
server_key_version: "encryptionkey@0"
I20260926 14:50:16.306036 8083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:16.306734 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:16.306926 8083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:16.307020 8083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "cd8f91be279347078b2db83d30caa870"
format_stamp: "Formatted at 2026-09-26 14:50:16 on dist-test-slave-s4p1"
server_key: "96b09e31c1ad46b394a37ec0865113e9"
server_key_iv: "cb66ba67c0b00234bf3aa05bd3f11dc0"
server_key_version: "encryptionkey@0"
I20260926 14:50:16.307072 8083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:16.321906 8083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:16.323683 8083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:16.323787 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:16.329710 8083 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:44829
I20260926 14:50:16.329764 8150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:44829 every 8 connection(s)
I20260926 14:50:16.330196 8083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:16.331193 8151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:16.333343 8151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870: Bootstrap starting.
I20260926 14:50:16.333868 8151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:16.334103 8151 log.cc:824] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:16.334692 8151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870: No bootstrap required, opened a new log
I20260926 14:50:16.336238 8151 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f91be279347078b2db83d30caa870" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44829 } }
I20260926 14:50:16.336373 8151 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:16.336396 8151 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8f91be279347078b2db83d30caa870, State: Initialized, Role: FOLLOWER
I20260926 14:50:16.336545 8151 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f91be279347078b2db83d30caa870" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44829 } }
I20260926 14:50:16.336623 8151 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:16.336644 8151 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:16.336676 8151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:16.337368 8151 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f91be279347078b2db83d30caa870" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44829 } }
I20260926 14:50:16.337450 8151 leader_election.cc:302] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd8f91be279347078b2db83d30caa870; no voters:
I20260926 14:50:16.337575 8151 leader_election.cc:288] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:16.337633 8154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:16.337832 8151 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:16.337821 8154 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [term 1 LEADER]: Becoming Leader. State: Replica: cd8f91be279347078b2db83d30caa870, State: Running, Role: LEADER
I20260926 14:50:16.338012 8154 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f91be279347078b2db83d30caa870" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44829 } }
I20260926 14:50:16.338078 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8083
I20260926 14:50:16.338161 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:16.338339 5182 external_mini_cluster.cc:1428] Setting key bc9ab41beb876c99be8954eaac7b39c3
I20260926 14:50:16.338605 8156 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd8f91be279347078b2db83d30caa870. Latest consensus state: current_term: 1 leader_uuid: "cd8f91be279347078b2db83d30caa870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f91be279347078b2db83d30caa870" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44829 } } }
I20260926 14:50:16.338699 8156 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:16.338933 8155 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd8f91be279347078b2db83d30caa870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8f91be279347078b2db83d30caa870" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44829 } } }
I20260926 14:50:16.339080 8155 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:16.339462 8163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:16.339905 8163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:16.341580 8163 catalog_manager.cc:1347] Generated new cluster ID: 2c8bd3e8780443199c1873d4b7ca1243
I20260926 14:50:16.341632 8163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:16 dist-test-slave-s4p1 krb5kdc[8070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:16.351177 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:16.338846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.5.15.190:44829)
0926 14:50:16.340514 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:16.340521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:16.340544 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:50:16.340625 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:16.340630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:16.340677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:16.340832 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:16.341600 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:16.342976 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:16.343956 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:16.344813 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:16.344899 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:16.347602 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:16.347606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:16.347606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:16.347616 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:16.349405 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:16.349754 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:16.349757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:16.349757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:16.349841 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:16.349992 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:16.349993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:16.349993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:16.350279 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:16.350383 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:16.350815 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 14:50:16.350935 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1624,"thread_start_us":62,"threads_started":1}
I20260926 14:50:16.352079 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:16.366252 8163 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:16.366804 8163 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:16.372164 8163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd8f91be279347078b2db83d30caa870: Generated new TSK 0
I20260926 14:50:16.372354 8163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:16.381268 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:16.377579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.5.15.190:44829)
0926 14:50:16.377636 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:16.377638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:16.377740 (+ 102us) server_negotiation.cc:407] Connection header received
0926 14:50:16.377806 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:16.377807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:16.377818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:16.377873 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:16.378205 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:16.379079 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:16.379776 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:16.380086 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:16.380150 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:16.380776 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:16.380777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:16.380777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:16.381027 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:16.381044 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:16.381051 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:16.381111 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:16.381173 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:50:16.381198 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:50:16.383616 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8083
2026-09-26T14:50:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1286 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:16 dist-test-slave-s4p1 krb5kdc[8184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:16 dist-test-slave-s4p1 krb5kdc[8184](info): set up 2 sockets
Sep 26 14:50:16 dist-test-slave-s4p1 krb5kdc[8184](info): commencing operation
krb5kdc: starting...
W20260926 14:50:17.386646 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:17.499696 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44917
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:17.578107 8197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:17.578312 8197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:17.578348 8197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:17.579445 8197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:17.579481 8197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:17.579495 8197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:17.579509 8197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:17.579520 8197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:17.580900 8197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:17.581092 8197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:17.581262 8197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:17.586066 8203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:17.586105 8202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:17.586114 8205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:17.586129 8197 server_base.cc:1034] running on GCE node
I20260926 14:50:17.586545 8197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:17.586905 8197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:17.588071 8197 hybrid_clock.cc:648] HybridClock initialized: now 1790434217588062 us; error 40 us; skew 500 ppm
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:17.590505 8197 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:17.591001 8197 webserver.cc:460] Webserver started at http://127.5.15.190:41165/ using document root <none> and password file <none>
I20260926 14:50:17.591248 8197 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:17.591317 8197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:17.591468 8197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:17.592514 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "42596df050bf455eb0a106f726552297"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "8ffd8319a3fc9396e99ac28dc2c53914"
server_key_iv: "fe2339a47e301a9a9d6e922199cd7fcb"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.592880 8197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "42596df050bf455eb0a106f726552297"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "8ffd8319a3fc9396e99ac28dc2c53914"
server_key_iv: "fe2339a47e301a9a9d6e922199cd7fcb"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.594193 8197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:17.595053 8212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.595253 8197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:17.595382 8197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "42596df050bf455eb0a106f726552297"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "8ffd8319a3fc9396e99ac28dc2c53914"
server_key_iv: "fe2339a47e301a9a9d6e922199cd7fcb"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.595460 8197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:17.609669 8197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:17.612102 8197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:17.612264 8197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:17.616689 8197 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:44917
I20260926 14:50:17.616743 8264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:44917 every 8 connection(s)
I20260926 14:50:17.617090 8197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:17.618060 8265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:17.620285 8265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297: Bootstrap starting.
I20260926 14:50:17.620805 8265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:17.621035 8265 log.cc:824] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:17.621706 8265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297: No bootstrap required, opened a new log
I20260926 14:50:17.623297 8265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42596df050bf455eb0a106f726552297" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44917 } }
I20260926 14:50:17.623426 8265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:17.623448 8265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42596df050bf455eb0a106f726552297, State: Initialized, Role: FOLLOWER
I20260926 14:50:17.623545 8265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42596df050bf455eb0a106f726552297" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44917 } }
I20260926 14:50:17.623627 8265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:17.623661 8265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:17.623733 8265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:17.624168 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8197
I20260926 14:50:17.624244 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:17.624423 5182 external_mini_cluster.cc:1428] Setting key a5d7a93389d6b9bcc3b0e8a7e8ef133e
I20260926 14:50:17.624711 8265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42596df050bf455eb0a106f726552297" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44917 } }
I20260926 14:50:17.624828 8265 leader_election.cc:302] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42596df050bf455eb0a106f726552297; no voters:
I20260926 14:50:17.624994 8265 leader_election.cc:288] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:17.625027 8269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:17.625238 8269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [term 1 LEADER]: Becoming Leader. State: Replica: 42596df050bf455eb0a106f726552297, State: Running, Role: LEADER
I20260926 14:50:17.625308 8265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:17.625352 8269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42596df050bf455eb0a106f726552297" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44917 } }
I20260926 14:50:17.625689 8269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42596df050bf455eb0a106f726552297. Latest consensus state: current_term: 1 leader_uuid: "42596df050bf455eb0a106f726552297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42596df050bf455eb0a106f726552297" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44917 } } }
I20260926 14:50:17.625782 8269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:17.625701 8271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42596df050bf455eb0a106f726552297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42596df050bf455eb0a106f726552297" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44917 } } }
I20260926 14:50:17.625906 8271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:17.626060 8277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:17.626559 8277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:17.628372 8277 catalog_manager.cc:1347] Generated new cluster ID: 71dd54ea2ffe49cf9aa0f6a0c6ad47d7
I20260926 14:50:17.628432 8277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:17.636091 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:17.624981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.5.15.190:44917)
0926 14:50:17.625158 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:17.625167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:17.625201 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:17.625313 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:17.625320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:17.625377 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:17.626386 (+ 1009us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:17.626527 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.627965 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.629043 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.629621 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.629720 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:17.632375 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:17.632379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:17.632379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:17.632391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:17.634459 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:17.634848 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:17.634851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:17.634852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:17.634967 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:17.635167 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:17.635168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:17.635168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:17.635482 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:17.635641 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:17.635739 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 14:50:17.635875 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260926 14:50:17.636469 8277 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:17.637123 8277 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:17.671394 8277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42596df050bf455eb0a106f726552297: Generated new TSK 0
I20260926 14:50:17.671640 8277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:17.699494 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44917
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:17.787436 8291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:17.787606 8291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:17.787627 8291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:17.788653 8291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:17.788690 8291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:17.788714 8291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:17.790025 8291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:17.790256 8291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:17.790480 8291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:17.795205 8299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:17.795234 8291 server_base.cc:1034] running on GCE node
W20260926 14:50:17.795205 8297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:17.795199 8296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:17.795763 8291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:17.796020 8291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:17.797219 8291 hybrid_clock.cc:648] HybridClock initialized: now 1790434217797194 us; error 37 us; skew 500 ppm
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:17.799887 8291 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:17.800273 8291 webserver.cc:460] Webserver started at http://127.5.15.129:41577/ using document root <none> and password file <none>
I20260926 14:50:17.800509 8291 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:17.800571 8291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:17.800706 8291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:17.801702 8291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "98cac684b74f464dad21c61e155c8af7"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "80b893514406e5def642a41ff187c68c"
server_key_iv: "1a81d6f4f039090f78a3062946581b85"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.802055 8291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "98cac684b74f464dad21c61e155c8af7"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "80b893514406e5def642a41ff187c68c"
server_key_iv: "1a81d6f4f039090f78a3062946581b85"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.803371 8291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 14:50:17.804073 8306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.804260 8291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:17.804337 8291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "98cac684b74f464dad21c61e155c8af7"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "80b893514406e5def642a41ff187c68c"
server_key_iv: "1a81d6f4f039090f78a3062946581b85"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.804399 8291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:17.815752 8291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:17.817476 8291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:17.817582 8291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:17.817765 8291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:17.817986 8291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:17.818015 8291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.818034 8291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:17.818046 8291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.823730 8291 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:43909
I20260926 14:50:17.823817 8419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:43909 every 8 connection(s)
I20260926 14:50:17.824184 8291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:17.824515 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8291
I20260926 14:50:17.824601 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:17.824800 5182 external_mini_cluster.cc:1428] Setting key aa92b97b6e2ccff4dc688e35dbadeca6
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
I20260926 14:50:17.833986 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:17.824981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:45825 (local address 127.5.15.190:44917)
0926 14:50:17.825029 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:17.825031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:17.825257 (+ 226us) server_negotiation.cc:407] Connection header received
0926 14:50:17.825851 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:17.825852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:17.825868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:17.825939 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:17.827177 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.827649 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.828686 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.829018 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.829091 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:17.831159 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:17.831160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:17.831161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:17.831169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:17.832633 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:17.833047 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:17.833048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:17.833048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:17.833106 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:17.833365 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:17.833365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:17.833366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:17.833474 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:17.833543 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:17.833810 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 14:50:17.833901 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 14:50:17.834158 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:17.824944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44917 (local address 127.5.15.129:45825)
0926 14:50:17.825126 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:17.825171 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:17.825752 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:17.825989 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:17.825992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:17.826741 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:17.827082 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:17.827089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.827739 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.827740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:17.828591 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:17.828606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.829099 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.829100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:17.829191 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:17.829730 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:17.829738 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:17.831066 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:17.832734 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:17.832740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:17.832742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:17.832959 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:17.833178 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:17.833179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:17.833179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:17.833287 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:17.833547 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:17.833554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:17.833684 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 14:50:17.833788 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:50:17.833936 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260926 14:50:17.834630 8420 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44917
I20260926 14:50:17.834710 8420 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:17.834896 8420 heartbeater.cc:511] Master 127.5.15.190:44917 requested a full tablet report, sending...
I20260926 14:50:17.835435 8229 ts_manager.cc:194] Registered new tserver with Master: 98cac684b74f464dad21c61e155c8af7 (127.5.15.129:43909)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
I20260926 14:50:17.836115 8229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:45825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:17.849830 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44917
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:17.929610 8425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:17.929800 8425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:17.929822 8425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:17.930866 8425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:17.930908 8425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:17.930933 8425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:17.932225 8425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:17.932399 8425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:17.932582 8425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:17.937438 8430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:17.937449 8431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:17.937443 8433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:17.937633 8425 server_base.cc:1034] running on GCE node
I20260926 14:50:17.937891 8425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:17.938122 8425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:17.939278 8425 hybrid_clock.cc:648] HybridClock initialized: now 1790434217939277 us; error 67 us; skew 500 ppm
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:17.941618 8425 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:17.942059 8425 webserver.cc:460] Webserver started at http://127.5.15.130:37399/ using document root <none> and password file <none>
I20260926 14:50:17.942302 8425 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:17.942377 8425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:17.942525 8425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:17.943580 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "ae79c536e01c4ce293db43a5d7755564"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "94fb5d4d6525dba535e0ce712ef98808"
server_key_iv: "a6373b27d6c57ec7c3056ec1819e6575"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.943948 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "ae79c536e01c4ce293db43a5d7755564"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "94fb5d4d6525dba535e0ce712ef98808"
server_key_iv: "a6373b27d6c57ec7c3056ec1819e6575"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.945245 8425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:17.945960 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.946122 8425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:17.946219 8425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "ae79c536e01c4ce293db43a5d7755564"
format_stamp: "Formatted at 2026-09-26 14:50:17 on dist-test-slave-s4p1"
server_key: "94fb5d4d6525dba535e0ce712ef98808"
server_key_iv: "a6373b27d6c57ec7c3056ec1819e6575"
server_key_version: "encryptionkey@0"
I20260926 14:50:17.946302 8425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:17.980693 8425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:17.982728 8425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:17.982915 8425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:17.983156 8425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:17.983480 8425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:17.983537 8425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.983578 8425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:17.983605 8425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:17.989418 8425 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:41487
I20260926 14:50:17.989466 8553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:41487 every 8 connection(s)
I20260926 14:50:17.989778 8425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:17.994601 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8425
I20260926 14:50:17.994717 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:17.994949 5182 external_mini_cluster.cc:1428] Setting key bed177674f0ff18f1fcae45b04d3a222
Sep 26 14:50:17 dist-test-slave-s4p1 krb5kdc[8184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434217, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:17.999199 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:17.990394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44917 (local address 127.5.15.130:34209)
0926 14:50:17.990565 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:17.990606 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:17.991144 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:17.991356 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:17.991359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:17.991947 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:17.992133 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:17.992137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.992783 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.992784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:17.993564 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:17.993567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.993876 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.993877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:17.993972 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:17.994415 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:17.994422 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:17.996053 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:17.997851 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:17.997857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:17.997858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:17.998060 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:17.998252 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:17.998253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:17.998253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:17.998345 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:17.998640 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:17.998645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:17.998847 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 14:50:17.998933 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 14:50:17.999047 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260926 14:50:17.999193 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:17.990445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:34209 (local address 127.5.15.190:44917)
0926 14:50:17.990490 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:17.990491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:17.990705 (+ 214us) server_negotiation.cc:407] Connection header received
0926 14:50:17.991238 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:17.991239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:17.991248 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:17.991304 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:17.992212 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.992683 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.993637 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:17.993807 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:17.993854 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:17.996158 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:17.996160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:17.996160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:17.996166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:17.997774 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:17.998135 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:17.998136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:17.998136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:17.998194 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:17.998412 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:17.998413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:17.998413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:17.998551 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:17.998623 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:17.998971 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 14:50:17.999058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:17.999918 8554 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44917
I20260926 14:50:18.000012 8554 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:18.000191 8554 heartbeater.cc:511] Master 127.5.15.190:44917 requested a full tablet report, sending...
I20260926 14:50:18.000547 8229 ts_manager.cc:194] Registered new tserver with Master: ae79c536e01c4ce293db43a5d7755564 (127.5.15.130:41487)
I20260926 14:50:18.000926 8229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:34209
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:18.021065 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44917
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:18.101418 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:18.101637 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:18.101658 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:18.102674 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:18.102708 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:18.102732 8559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:18.104112 8559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:18.104277 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:18.104445 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:18.109476 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:18.109503 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:18.109426 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:18.109596 8559 server_base.cc:1034] running on GCE node
I20260926 14:50:18.109823 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:18.110056 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:18.111214 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790434218111200 us; error 22 us; skew 500 ppm
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:18.113648 8559 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:18.114063 8559 webserver.cc:460] Webserver started at http://127.5.15.131:34871/ using document root <none> and password file <none>
I20260926 14:50:18.114283 8559 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:18.114348 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:18.114509 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:18.115578 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "ec61709300cc4c4b96393a1127d3c2cc"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-s4p1"
server_key: "e7042ce97f0a6df1188b1ca024abf4d7"
server_key_iv: "50d9d25bdf3cc6f2704b7a7f4f5b2cc1"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.115943 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "ec61709300cc4c4b96393a1127d3c2cc"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-s4p1"
server_key: "e7042ce97f0a6df1188b1ca024abf4d7"
server_key_iv: "50d9d25bdf3cc6f2704b7a7f4f5b2cc1"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.117206 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:50:18.117895 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:18.118064 8559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:18.118160 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "ec61709300cc4c4b96393a1127d3c2cc"
format_stamp: "Formatted at 2026-09-26 14:50:18 on dist-test-slave-s4p1"
server_key: "e7042ce97f0a6df1188b1ca024abf4d7"
server_key_iv: "50d9d25bdf3cc6f2704b7a7f4f5b2cc1"
server_key_version: "encryptionkey@0"
I20260926 14:50:18.118243 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:18.135682 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:18.137892 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:18.138054 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:18.138266 8559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:18.138571 8559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:18.138625 8559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:18.138664 8559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:18.138690 8559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:18.144327 8559 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:34505
I20260926 14:50:18.144400 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:34505 every 8 connection(s)
I20260926 14:50:18.144704 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:18.145408 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8559
I20260926 14:50:18.145501 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:18.145706 5182 external_mini_cluster.cc:1428] Setting key cd2e06c3552047db32a1368a0e81defd
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:18.153956 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:18.145432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:54339 (local address 127.5.15.190:44917)
0926 14:50:18.145500 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:18.145502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:18.145701 (+ 199us) server_negotiation.cc:407] Connection header received
0926 14:50:18.146276 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:18.146277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:18.146289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:18.146344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:18.147346 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.147842 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.148921 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.149132 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.149211 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:18.151325 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:18.151326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:18.151327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:18.151340 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:18.152588 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:18.152997 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:18.152999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:18.152999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:18.153059 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:18.153304 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:18.153305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:18.153305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:18.153440 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:18.153517 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:18.153786 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 14:50:18.153869 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:18.154127 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:18.145389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44917 (local address 127.5.15.131:54339)
0926 14:50:18.145566 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:18.145606 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:18.146177 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:18.146411 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:18.146416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:18.147045 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:18.147252 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:18.147257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.147940 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.147941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:18.148840 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:18.148843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.149220 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.149221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:18.149351 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:18.149850 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:18.149860 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:18.151235 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:18.152666 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:18.152672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:18.152674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:18.152901 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:18.153133 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:18.153134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:18.153134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:18.153239 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:18.153508 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:18.153514 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:18.153648 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 14:50:18.153750 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:50:18.153899 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260926 14:50:18.154677 8688 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44917
I20260926 14:50:18.154798 8688 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:18.154973 8688 heartbeater.cc:511] Master 127.5.15.190:44917 requested a full tablet report, sending...
I20260926 14:50:18.155318 8229 ts_manager.cc:194] Registered new tserver with Master: ec61709300cc4c4b96393a1127d3c2cc (127.5.15.131:34505)
I20260926 14:50:18.155668 8229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:54339
I20260926 14:50:18.157177 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434218, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:18.178331 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:18.171538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.5.15.190:44917)
0926 14:50:18.171605 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:18.171607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:18.171686 (+ 79us) server_negotiation.cc:407] Connection header received
0926 14:50:18.171725 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:18.171726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:18.171734 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:18.171789 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:18.172296 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.172747 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.173411 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:18.173575 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:18.173621 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:18.175745 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:18.175746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:18.175746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:18.175754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:18.177125 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:18.177515 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:18.177516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:18.177517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:18.177572 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:18.177788 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:18.177789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:18.177789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:18.177932 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:18.178035 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:18.178147 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 14:50:18.178222 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 14:50:18.178661 8229 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44666
I20260926 14:50:18.179148 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8291
I20260926 14:50:18.183846 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8425
I20260926 14:50:18.187932 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8559
I20260926 14:50:18.191808 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8197
2026-09-26T14:50:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1810 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8700](info): set up 2 sockets
Sep 26 14:50:18 dist-test-slave-s4p1 krb5kdc[8700](info): commencing operation
krb5kdc: starting...
W20260926 14:50:19.218580 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:19.328895 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36009
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:19.407234 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:19.407464 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:19.407500 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:19.408635 8713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:19.408699 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:19.408722 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:19.408752 8713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:19.408774 8713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:19.410122 8713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:19.410365 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:19.410564 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:19.415316 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.415407 8713 server_base.cc:1034] running on GCE node
W20260926 14:50:19.415325 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.415325 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.415758 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:19.416054 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:19.417244 8713 hybrid_clock.cc:648] HybridClock initialized: now 1790434219417238 us; error 37 us; skew 500 ppm
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:19.419866 8713 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:19.420276 8713 webserver.cc:460] Webserver started at http://127.5.15.190:44757/ using document root <none> and password file <none>
I20260926 14:50:19.420523 8713 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:19.420593 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:19.420742 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:19.421759 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "891482262a404b069feb52a693fd50d0"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "68434072a2e7c78a9985173fd6acfcac"
server_key_iv: "80f94967d0183125f28fa3633ef5b612"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.422137 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "891482262a404b069feb52a693fd50d0"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "68434072a2e7c78a9985173fd6acfcac"
server_key_iv: "80f94967d0183125f28fa3633ef5b612"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.423528 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:19.424212 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.424407 8713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.424515 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "891482262a404b069feb52a693fd50d0"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "68434072a2e7c78a9985173fd6acfcac"
server_key_iv: "80f94967d0183125f28fa3633ef5b612"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.424621 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.436026 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.437750 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.437942 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.441747 8713 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:36009
I20260926 14:50:19.441777 8780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:36009 every 8 connection(s)
I20260926 14:50:19.442239 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:19.442965 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8713
I20260926 14:50:19.442973 8781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:19.443073 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:19.443250 5182 external_mini_cluster.cc:1428] Setting key 42696a5888cdeda0b3af3d15fc86d686
I20260926 14:50:19.446024 8781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0: Bootstrap starting.
I20260926 14:50:19.446775 8781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:19.447072 8781 log.cc:824] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:19.447799 8781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0: No bootstrap required, opened a new log
I20260926 14:50:19.449720 8781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891482262a404b069feb52a693fd50d0" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36009 } }
I20260926 14:50:19.449888 8781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:19.449923 8781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891482262a404b069feb52a693fd50d0, State: Initialized, Role: FOLLOWER
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:19.450088 8781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891482262a404b069feb52a693fd50d0" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36009 } }
I20260926 14:50:19.450179 8781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:19.450227 8781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:19.450275 8781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:19.450841 8781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891482262a404b069feb52a693fd50d0" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36009 } }
I20260926 14:50:19.450948 8781 leader_election.cc:302] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891482262a404b069feb52a693fd50d0; no voters:
I20260926 14:50:19.451109 8781 leader_election.cc:288] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:19.451177 8786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:19.451308 8786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [term 1 LEADER]: Becoming Leader. State: Replica: 891482262a404b069feb52a693fd50d0, State: Running, Role: LEADER
I20260926 14:50:19.451352 8781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:19.451699 8786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891482262a404b069feb52a693fd50d0" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36009 } }
I20260926 14:50:19.452032 8786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891482262a404b069feb52a693fd50d0. Latest consensus state: current_term: 1 leader_uuid: "891482262a404b069feb52a693fd50d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891482262a404b069feb52a693fd50d0" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36009 } } }
I20260926 14:50:19.452250 8786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:19.452183 8788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891482262a404b069feb52a693fd50d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891482262a404b069feb52a693fd50d0" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36009 } } }
I20260926 14:50:19.452629 8788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:19.452736 8795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:19.453246 8795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:19.454731 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.443736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.5.15.190:36009)
0926 14:50:19.443902 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.443911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.443943 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:19.444380 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.444386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.444432 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.444540 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.445332 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.446260 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.447379 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.447886 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.448012 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.450470 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.450476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.450481 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.450491 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.452562 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.453035 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.453037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.453038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.453137 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.453377 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.453378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.453378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.453750 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.454267 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.454335 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.454473 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260926 14:50:19.455055 8795 catalog_manager.cc:1347] Generated new cluster ID: 928dd95ebbb84bf9bf643c5fffaa8285
I20260926 14:50:19.455106 8795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:19.469607 8795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:19.470129 8795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:19.476166 8795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891482262a404b069feb52a693fd50d0: Generated new TSK 0
I20260926 14:50:19.476372 8795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:19.502154 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36009
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:19.580202 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:19.580389 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:19.580411 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:19.581547 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:19.581593 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:19.581619 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:19.583024 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:19.583189 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:19.583374 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:19.588204 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.588173 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.588420 8807 server_base.cc:1034] running on GCE node
W20260926 14:50:19.588177 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.588721 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:19.588977 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:19.590121 8807 hybrid_clock.cc:648] HybridClock initialized: now 1790434219590113 us; error 30 us; skew 500 ppm
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:19.592495 8807 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:19.592905 8807 webserver.cc:460] Webserver started at http://127.5.15.129:44557/ using document root <none> and password file <none>
I20260926 14:50:19.593132 8807 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:19.593199 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:19.593343 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:19.594324 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "41140011e023481794f67ca83e8479ac"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "1aa4e4b774a9f62e3bc314f0fe565898"
server_key_iv: "7783c621ddad54270b2c973208202f08"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.594740 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "41140011e023481794f67ca83e8479ac"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "1aa4e4b774a9f62e3bc314f0fe565898"
server_key_iv: "7783c621ddad54270b2c973208202f08"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.596096 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:19.596823 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.597028 8807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:19.597119 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "41140011e023481794f67ca83e8479ac"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "1aa4e4b774a9f62e3bc314f0fe565898"
server_key_iv: "7783c621ddad54270b2c973208202f08"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.597194 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.604225 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.605865 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.606016 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.606273 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:19.606561 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:19.606628 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.606667 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:19.606694 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.612639 8807 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:46257
I20260926 14:50:19.612684 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:46257 every 8 connection(s)
I20260926 14:50:19.612982 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:19.616446 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8807
I20260926 14:50:19.616519 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:19.616689 5182 external_mini_cluster.cc:1428] Setting key 308ece9d5e83dc0411e93edad47c72b2
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:19.622859 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.613719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:45473 (local address 127.5.15.190:36009)
0926 14:50:19.613775 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.613777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.613989 (+ 212us) server_negotiation.cc:407] Connection header received
0926 14:50:19.614928 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.614929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.614938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.614983 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.615523 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.616018 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.617167 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.617484 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.617555 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.619999 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.620001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.620001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.620007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.621385 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.621853 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.621854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.621855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.621899 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.622160 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.622160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.622160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.622257 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.622310 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.622657 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.622719 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:19.623100 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.613701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36009 (local address 127.5.15.129:45473)
0926 14:50:19.613859 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:19.613905 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:19.614824 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:19.615032 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:19.615035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:19.615260 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:19.615436 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.615439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.616089 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.616090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.617102 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.617104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.617568 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.617569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.617663 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.618172 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:19.618180 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:19.619900 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:19.621467 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.621474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.621476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.621782 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.621940 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.621941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.621941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.622096 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.622322 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:19.622329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:19.622509 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 14:50:19.622645 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 14:50:19.622860 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260926 14:50:19.623646 8936 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36009
I20260926 14:50:19.623737 8936 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:19.623934 8936 heartbeater.cc:511] Master 127.5.15.190:36009 requested a full tablet report, sending...
I20260926 14:50:19.624420 8745 ts_manager.cc:194] Registered new tserver with Master: 41140011e023481794f67ca83e8479ac (127.5.15.129:46257)
I20260926 14:50:19.624986 8745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:45473
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:19.641805 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36009
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:19.722033 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:19.722275 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:19.722316 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:19.723461 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:19.723526 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:19.723569 8941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:19.725059 8941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:19.725337 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:19.725561 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:19.730700 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.730716 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.730809 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.730913 8941 server_base.cc:1034] running on GCE node
I20260926 14:50:19.731164 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:19.731371 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:19.732532 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790434219732511 us; error 37 us; skew 500 ppm
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:19.734910 8941 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:19.735270 8941 webserver.cc:460] Webserver started at http://127.5.15.130:33155/ using document root <none> and password file <none>
I20260926 14:50:19.735436 8941 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:19.735477 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:19.735565 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:19.736526 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "a65acf2a188c4866a711c1f61925ea98"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "f63e32be1616bbd1f9bde086a3a198da"
server_key_iv: "628f1aa0afe3eb7b654fc03c06e758cb"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.736824 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "a65acf2a188c4866a711c1f61925ea98"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "f63e32be1616bbd1f9bde086a3a198da"
server_key_iv: "628f1aa0afe3eb7b654fc03c06e758cb"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.738008 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:19.738670 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.738842 8941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:19.738936 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "a65acf2a188c4866a711c1f61925ea98"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "f63e32be1616bbd1f9bde086a3a198da"
server_key_iv: "628f1aa0afe3eb7b654fc03c06e758cb"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.738991 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.747119 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.749310 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.749421 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.749584 8941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:19.749853 8941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:19.749888 8941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.749909 8941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:19.749966 8941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.755789 8941 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:40977
I20260926 14:50:19.755872 9069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:40977 every 8 connection(s)
I20260926 14:50:19.756186 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:19.764706 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.756959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:35621 (local address 127.5.15.190:36009)
0926 14:50:19.757037 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.757039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.757199 (+ 160us) server_negotiation.cc:407] Connection header received
0926 14:50:19.758069 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.758071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.758080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.758130 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.758698 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.759274 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.760201 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.760382 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.760429 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.762478 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.762480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.762480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.762487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.763657 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.763992 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.763993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.763993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.764038 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.764230 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.764231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.764231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.764320 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.764380 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.764594 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.764648 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:50:19.764858 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.756915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36009 (local address 127.5.15.130:35621)
0926 14:50:19.757082 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:19.757117 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:19.757982 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:19.758185 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:19.758187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:19.758450 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:19.758625 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.758629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.759356 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.759357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.760127 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.760129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.760450 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.760451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.760542 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.760951 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:19.760959 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:19.762390 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:19.763722 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.763728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.763729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.763929 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.764091 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.764092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.764092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.764180 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.764386 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:19.764391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:19.764492 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 14:50:19.764568 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:50:19.764674 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260926 14:50:19.765319 9070 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36009
I20260926 14:50:19.765424 9070 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:19.765607 9070 heartbeater.cc:511] Master 127.5.15.190:36009 requested a full tablet report, sending...
I20260926 14:50:19.765970 8745 ts_manager.cc:194] Registered new tserver with Master: a65acf2a188c4866a711c1f61925ea98 (127.5.15.130:40977)
I20260926 14:50:19.766185 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 8941
I20260926 14:50:19.766286 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:19.766423 8745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:35621
I20260926 14:50:19.766474 5182 external_mini_cluster.cc:1428] Setting key dc1418943c3c91fbd397caac898bb2f0
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:19.789860 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36009
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:19.868968 9075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:19.869155 9075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:19.869177 9075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:19.870268 9075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:19.870306 9075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:19.870330 9075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:19.871718 9075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:19.871961 9075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:19.872174 9075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:19.877003 9081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.877070 9080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:19.877012 9083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:19.877151 9075 server_base.cc:1034] running on GCE node
I20260926 14:50:19.877530 9075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:19.877761 9075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:19.878913 9075 hybrid_clock.cc:648] HybridClock initialized: now 1790434219878908 us; error 34 us; skew 500 ppm
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:19.881228 9075 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:19.881630 9075 webserver.cc:460] Webserver started at http://127.5.15.131:45315/ using document root <none> and password file <none>
I20260926 14:50:19.881839 9075 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:19.881903 9075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:19.882047 9075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:19.883072 9075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "6dd152ee8123484ca621757d8cbaa7db"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "17d22a39ce4591e47f42b212bcb517c1"
server_key_iv: "9b99f9b3f6ec80f0c29b9b965baa0c0e"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.883399 9075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "6dd152ee8123484ca621757d8cbaa7db"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "17d22a39ce4591e47f42b212bcb517c1"
server_key_iv: "9b99f9b3f6ec80f0c29b9b965baa0c0e"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.884688 9075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:19.885347 9090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.885510 9075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:19.885633 9075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "6dd152ee8123484ca621757d8cbaa7db"
format_stamp: "Formatted at 2026-09-26 14:50:19 on dist-test-slave-s4p1"
server_key: "17d22a39ce4591e47f42b212bcb517c1"
server_key_iv: "9b99f9b3f6ec80f0c29b9b965baa0c0e"
server_key_version: "encryptionkey@0"
I20260926 14:50:19.885710 9075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:19.894822 9075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:19.896503 9075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:19.896658 9075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:19.896879 9075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:19.897166 9075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:19.897217 9075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.897249 9075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:19.897274 9075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:19.902895 9075 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:41189
I20260926 14:50:19.902930 9203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:41189 every 8 connection(s)
I20260926 14:50:19.903231 9075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:19.904280 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9075
I20260926 14:50:19.904364 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:19.904537 5182 external_mini_cluster.cc:1428] Setting key 3df80013e46fbbce55689838969f3deb
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:19.911854 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.903945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:52859 (local address 127.5.15.190:36009)
0926 14:50:19.904010 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.904012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.904173 (+ 161us) server_negotiation.cc:407] Connection header received
0926 14:50:19.905297 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.905298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.905310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.905405 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.905986 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.906475 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.907484 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.907660 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.907711 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.909534 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.909535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.909536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.909542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.910706 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.911133 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.911134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.911134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.911179 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.911378 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.911378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.911379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.911471 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.911519 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.911727 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.911789 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:19.912024 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.903908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36009 (local address 127.5.15.131:52859)
0926 14:50:19.904051 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:19.904088 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:19.905214 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:19.905478 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:19.905481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:19.905737 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:19.905914 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.905918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.906550 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.906551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.907410 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:19.907412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.907730 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.907730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:19.907820 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.908213 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:19.908221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:19.909452 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:19.910834 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.910840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.910842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.911073 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.911236 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:19.911236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:19.911237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:19.911323 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:19.911523 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:19.911526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:19.911624 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:50:19.911700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:50:19.911804 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260926 14:50:19.912492 9204 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36009
I20260926 14:50:19.912585 9204 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:19.912762 9204 heartbeater.cc:511] Master 127.5.15.190:36009 requested a full tablet report, sending...
I20260926 14:50:19.913166 8745 ts_manager.cc:194] Registered new tserver with Master: 6dd152ee8123484ca621757d8cbaa7db (127.5.15.131:41189)
I20260926 14:50:19.913532 8745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:52859
I20260926 14:50:19.916184 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:19.938035 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.931019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.5.15.129:46257)
0926 14:50:19.931198 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.931202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.931211 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:19.931236 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.931237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.931250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.931313 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.931817 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.932494 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.933142 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.933468 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.933518 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.935579 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.935587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.935587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.935593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.937156 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.937443 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.937445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.937446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.937498 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.937637 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.937637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.937637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.937748 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.937868 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.937901 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.937959 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
Sep 26 14:50:19 dist-test-slave-s4p1 krb5kdc[8700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:19.945822 8784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:19.939346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.5.15.190:36009)
0926 14:50:19.939409 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:19.939411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:19.939421 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:19.939497 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:19.939498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:19.939508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:19.939579 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:19.940008 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.940477 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.941161 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:19.941350 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:19.941445 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:19.943467 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:19.943468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:19.943469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:19.943475 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:19.944590 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.944939 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.944940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.944941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.945015 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:19.945217 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:19.945218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:19.945218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:19.945386 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:19.945496 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:19.945642 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 14:50:19.945717 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 14:50:19.946030 8745 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42976
I20260926 14:50:19.946341 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8807
I20260926 14:50:19.951579 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8941
I20260926 14:50:19.956017 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9075
I20260926 14:50:19.959810 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 8713
2026-09-26T14:50:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1768 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:20 dist-test-slave-s4p1 krb5kdc[9216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:20 dist-test-slave-s4p1 krb5kdc[9216](info): set up 2 sockets
Sep 26 14:50:20 dist-test-slave-s4p1 krb5kdc[9216](info): commencing operation
krb5kdc: starting...
W20260926 14:50:20.978634 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:21.090045 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41105
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:41105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:21.169989 9229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:21.170236 9229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:21.170311 9229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:21.171487 9229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:21.171554 9229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:21.171577 9229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:21.171597 9229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:21.171626 9229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:21.173087 9229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:41105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:21.173348 9229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:21.173583 9229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:21.178212 9235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.178220 9234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.178282 9237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.178427 9229 server_base.cc:1034] running on GCE node
I20260926 14:50:21.178700 9229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:21.179042 9229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:21.180186 9229 hybrid_clock.cc:648] HybridClock initialized: now 1790434221180166 us; error 35 us; skew 500 ppm
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:21.182524 9229 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:21.182917 9229 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:21.182965 9229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:21.183111 9229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:21.184123 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "6129666498de4d4bb78eb731194db0ed"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "27170dab39e39d3ef54988d5dc0787f2"
server_key_iv: "3371e0908374ad7e42e691bdd9ce4872"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.184468 9229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "6129666498de4d4bb78eb731194db0ed"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "27170dab39e39d3ef54988d5dc0787f2"
server_key_iv: "3371e0908374ad7e42e691bdd9ce4872"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.185696 9229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:21.186379 9243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.186568 9229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:21.186637 9229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "6129666498de4d4bb78eb731194db0ed"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "27170dab39e39d3ef54988d5dc0787f2"
server_key_iv: "3371e0908374ad7e42e691bdd9ce4872"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.186725 9229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:21.196938 9229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:21.198580 9229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:21.198730 9229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:21.202533 9229 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:41105
I20260926 14:50:21.202553 9295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:41105 every 8 connection(s)
I20260926 14:50:21.202874 9229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:21.203715 9296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:21.204221 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9229
I20260926 14:50:21.204304 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:21.204490 5182 external_mini_cluster.cc:1428] Setting key 0d3d278113c9b714df63a2fff62dadd8
I20260926 14:50:21.206543 9296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed: Bootstrap starting.
I20260926 14:50:21.207237 9296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:21.207522 9296 log.cc:824] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:21.208163 9296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed: No bootstrap required, opened a new log
I20260926 14:50:21.210168 9296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6129666498de4d4bb78eb731194db0ed" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41105 } }
I20260926 14:50:21.210289 9296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:21.210310 9296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6129666498de4d4bb78eb731194db0ed, State: Initialized, Role: FOLLOWER
I20260926 14:50:21.210438 9296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6129666498de4d4bb78eb731194db0ed" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41105 } }
I20260926 14:50:21.210510 9296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:21.210533 9296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:21.210597 9296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:21.211284 9296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6129666498de4d4bb78eb731194db0ed" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41105 } }
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:21.211392 9296 leader_election.cc:302] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6129666498de4d4bb78eb731194db0ed; no voters:
I20260926 14:50:21.211582 9296 leader_election.cc:288] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:21.211664 9301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:21.211805 9301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [term 1 LEADER]: Becoming Leader. State: Replica: 6129666498de4d4bb78eb731194db0ed, State: Running, Role: LEADER
I20260926 14:50:21.211841 9296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:21.211911 9301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6129666498de4d4bb78eb731194db0ed" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41105 } }
I20260926 14:50:21.212224 9301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6129666498de4d4bb78eb731194db0ed. Latest consensus state: current_term: 1 leader_uuid: "6129666498de4d4bb78eb731194db0ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6129666498de4d4bb78eb731194db0ed" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41105 } } }
I20260926 14:50:21.212319 9301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:21.212525 9307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:21.212247 9302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6129666498de4d4bb78eb731194db0ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6129666498de4d4bb78eb731194db0ed" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41105 } } }
I20260926 14:50:21.212752 9302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:21.212986 9307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:21.215452 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.204980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.5.15.190:41105)
0926 14:50:21.205145 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.205154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.205190 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:50:21.205730 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.205738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.205795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.205955 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:21.206501 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.207855 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.208638 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.209261 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.209367 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.212070 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:21.212073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:21.212074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:21.212083 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:21.214024 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.214355 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.214358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.214358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.214450 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.214649 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.214650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.214650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.214939 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:21.215099 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.215164 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.215286 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260926 14:50:21.219446 9307 catalog_manager.cc:1347] Generated new cluster ID: 71e36bc7bced4c72a5be3f09e82de60e
I20260926 14:50:21.219503 9307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:21.233749 9307 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:21.234477 9307 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:21.237643 9307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed: Generated new TSK 0
I20260926 14:50:21.237788 9307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:21.267624 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41105
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:21.347203 9322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:21.347389 9322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:21.347412 9322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:21.348564 9322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:21.348603 9322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:21.348629 9322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:21.350085 9322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:21.350252 9322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:21.350507 9322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:21.355302 9327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.355279 9328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.355279 9330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.355494 9322 server_base.cc:1034] running on GCE node
I20260926 14:50:21.355700 9322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:21.355927 9322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:21.357112 9322 hybrid_clock.cc:648] HybridClock initialized: now 1790434221357108 us; error 42 us; skew 500 ppm
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:21.359546 9322 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:21.359900 9322 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:21.359975 9322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:21.360116 9322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:21.361114 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "8b7420dd8bc543a6ab8725f61ae1b595"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "aee38a3e86870bcbc0c0aeb3f2167396"
server_key_iv: "ea1faa78a2f30dde53f36577cfd577da"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.361471 9322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "8b7420dd8bc543a6ab8725f61ae1b595"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "aee38a3e86870bcbc0c0aeb3f2167396"
server_key_iv: "ea1faa78a2f30dde53f36577cfd577da"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.362710 9322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:21.363427 9336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.363588 9322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:21.363698 9322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "8b7420dd8bc543a6ab8725f61ae1b595"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "aee38a3e86870bcbc0c0aeb3f2167396"
server_key_iv: "ea1faa78a2f30dde53f36577cfd577da"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.363785 9322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:21.380482 9322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:21.382365 9322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:21.382525 9322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:21.382830 9322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:21.383145 9322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:21.383199 9322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.383240 9322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:21.383267 9322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.389228 9322 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:37619
I20260926 14:50:21.389271 9449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:37619 every 8 connection(s)
I20260926 14:50:21.389554 9322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:21.392133 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9322
I20260926 14:50:21.392235 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:21.392397 5182 external_mini_cluster.cc:1428] Setting key 84c9a014acad21e1eaea8499d83c59bc
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:21.399893 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.390209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:37521 (local address 127.5.15.190:41105)
0926 14:50:21.390256 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.390258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.390481 (+ 223us) server_negotiation.cc:407] Connection header received
0926 14:50:21.390986 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.390987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.390997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.391046 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:21.391873 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.392645 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.393753 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.394089 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.394159 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.396939 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:21.396940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:21.396940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:21.396946 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:21.398385 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.398930 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.398931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.398931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.398982 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.399279 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.399280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.399280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.399376 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:21.399430 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.399766 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.399832 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:21.400139 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.390206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41105 (local address 127.5.15.129:37521)
0926 14:50:21.390344 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.390378 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.390887 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.391097 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.391100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.391341 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:21.391801 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.391807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.392740 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.392741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.393664 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.393666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.394180 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.394181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.394338 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.394948 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:21.394959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:21.396848 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:21.398478 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:21.398487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:21.398490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:21.398856 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:21.399048 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:21.399049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:21.399049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:21.399198 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:21.399438 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:21.399444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:21.399617 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.399731 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.399902 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260926 14:50:21.400633 9450 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41105
I20260926 14:50:21.400738 9450 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:21.400883 9450 heartbeater.cc:511] Master 127.5.15.190:41105 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
I20260926 14:50:21.401422 9260 ts_manager.cc:194] Registered new tserver with Master: 8b7420dd8bc543a6ab8725f61ae1b595 (127.5.15.129:37619)
I20260926 14:50:21.402015 9260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:37521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:21.417192 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41105
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:21.503185 9455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:21.503484 9455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:21.503509 9455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:21.504635 9455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:21.504676 9455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:21.504731 9455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:21.506198 9455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:21.506431 9455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:21.506673 9455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:21.511415 9460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.511546 9463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.511698 9455 server_base.cc:1034] running on GCE node
W20260926 14:50:21.511446 9461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.511937 9455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:21.512178 9455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:21.513355 9455 hybrid_clock.cc:648] HybridClock initialized: now 1790434221513341 us; error 46 us; skew 500 ppm
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:21.515831 9455 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:21.516173 9455 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:21.516222 9455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:21.516309 9455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:21.517273 9455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "a6ed5486fdf2412196e25d8d3aebde1f"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "3a7d226713415d6869fdb33f5810079f"
server_key_iv: "2151e07f57805e7a057ac5dc98793fde"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.517584 9455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "a6ed5486fdf2412196e25d8d3aebde1f"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "3a7d226713415d6869fdb33f5810079f"
server_key_iv: "2151e07f57805e7a057ac5dc98793fde"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.518829 9455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:21.519481 9469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.519654 9455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:21.519721 9455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "a6ed5486fdf2412196e25d8d3aebde1f"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "3a7d226713415d6869fdb33f5810079f"
server_key_iv: "2151e07f57805e7a057ac5dc98793fde"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.519771 9455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:21.546795 9455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:21.549146 9455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:21.549273 9455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:21.549451 9455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:21.549780 9455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:21.549813 9455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.549863 9455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:21.549889 9455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.555847 9455 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:44861
I20260926 14:50:21.555883 9582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:44861 every 8 connection(s)
I20260926 14:50:21.556138 9455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:21.562016 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9455
I20260926 14:50:21.562131 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:21.562311 5182 external_mini_cluster.cc:1428] Setting key 1057084d396b774243d79915723a2db5
I20260926 14:50:21.565377 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.556790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:49223 (local address 127.5.15.190:41105)
0926 14:50:21.556852 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.556854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.557045 (+ 191us) server_negotiation.cc:407] Connection header received
0926 14:50:21.557556 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.557558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.557570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.557625 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:21.558559 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.559146 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.560139 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.560351 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.560427 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.562539 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:21.562541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:21.562541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:21.562550 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:21.564014 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.564428 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.564429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.564430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.564491 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.564742 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.564743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.564743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.564861 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:21.564938 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.565210 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.565285 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:21.565588 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.556730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41105 (local address 127.5.15.130:49223)
0926 14:50:21.556894 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.556943 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.557453 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.557704 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.557708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.557952 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:21.558452 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.558457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.559242 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.559243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.560046 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.560048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.560437 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.560438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.560566 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.560966 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:21.560974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:21.562402 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:21.564115 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:21.564122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:21.564124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:21.564349 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:21.564559 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:21.564560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:21.564560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:21.564674 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:21.564929 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:21.564935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:21.565075 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.565189 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.565353 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260926 14:50:21.566179 9583 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41105
I20260926 14:50:21.566275 9583 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:21.566426 9583 heartbeater.cc:511] Master 127.5.15.190:41105 requested a full tablet report, sending...
I20260926 14:50:21.566872 9260 ts_manager.cc:194] Registered new tserver with Master: a6ed5486fdf2412196e25d8d3aebde1f (127.5.15.130:44861)
I20260926 14:50:21.567236 9260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:49223
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:21.585392 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41105
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:40677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:21.664535 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:21.664726 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:21.664748 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:21.665805 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:21.665838 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:21.665864 9588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:21.667430 9588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:40677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:21.667649 9588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:21.667881 9588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:21.672647 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.672677 9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:21.672753 9596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:21.673019 9588 server_base.cc:1034] running on GCE node
I20260926 14:50:21.673203 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:21.673418 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:21.674551 9588 hybrid_clock.cc:648] HybridClock initialized: now 1790434221674540 us; error 27 us; skew 500 ppm
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:21.676990 9588 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:21.677263 9588 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:21.677304 9588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:21.677393 9588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:21.678350 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "1b45af04aee34f8f80cfb14affdd3b30"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "503ce69553dde0f33ea108a9c5ad8385"
server_key_iv: "ea9e039874b67bf7b9da02ba880ea938"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.678632 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "1b45af04aee34f8f80cfb14affdd3b30"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "503ce69553dde0f33ea108a9c5ad8385"
server_key_iv: "ea9e039874b67bf7b9da02ba880ea938"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.679867 9588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:21.680496 9602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.680683 9588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:21.680771 9588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "1b45af04aee34f8f80cfb14affdd3b30"
format_stamp: "Formatted at 2026-09-26 14:50:21 on dist-test-slave-s4p1"
server_key: "503ce69553dde0f33ea108a9c5ad8385"
server_key_iv: "ea9e039874b67bf7b9da02ba880ea938"
server_key_version: "encryptionkey@0"
I20260926 14:50:21.680850 9588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:21.693565 9588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:21.695847 9588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:21.696010 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:21.696237 9588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:21.696542 9588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:21.696574 9588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.696616 9588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:21.696643 9588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:21.702373 9588 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:33333
I20260926 14:50:21.702440 9715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:33333 every 8 connection(s)
I20260926 14:50:21.702689 9588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:21 dist-test-slave-s4p1 krb5kdc[9216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434221, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:21.709841 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9588
I20260926 14:50:21.709954 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:21.710129 5182 external_mini_cluster.cc:1428] Setting key 7a16ccbf79f7cad9148b2283ef87a9af
I20260926 14:50:21.712024 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.703418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:54791 (local address 127.5.15.190:41105)
0926 14:50:21.703502 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.703505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.703652 (+ 147us) server_negotiation.cc:407] Connection header received
0926 14:50:21.704155 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.704157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.704168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.704222 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:21.705148 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.705670 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.706665 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.706961 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.707043 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.709092 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:21.709095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:21.709095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:21.709102 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:21.710586 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.711022 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.711023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.711023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.711079 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.711344 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.711344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.711345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.711456 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:21.711526 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.711826 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.711898 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:50:21.712221 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.703383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41105 (local address 127.5.15.131:54791)
0926 14:50:21.703538 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.703576 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.704051 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.704294 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.704297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.704543 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:21.705043 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.705049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.705764 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.705766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.706569 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.706571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.707054 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.707055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.707171 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.707671 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:21.707679 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:21.708998 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:21.710678 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:21.710684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:21.710686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:21.710947 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:21.711153 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:21.711154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:21.711155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:21.711265 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:21.711517 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:21.711523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:21.711689 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.711805 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.711959 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260926 14:50:21.712718 9716 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41105
I20260926 14:50:21.712790 9716 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:21.712898 9716 heartbeater.cc:511] Master 127.5.15.190:41105 requested a full tablet report, sending...
I20260926 14:50:21.713217 9260 ts_manager.cc:194] Registered new tserver with Master: 1b45af04aee34f8f80cfb14affdd3b30 (127.5.15.131:33333)
I20260926 14:50:21.713624 9260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:54791
I20260926 14:50:21.721930 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:21.728499 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.723185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.5.15.190:41105)
0926 14:50:21.723220 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.723222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.723326 (+ 104us) server_negotiation.cc:407] Connection header received
0926 14:50:21.723402 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.723403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.723411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.723458 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:21.723934 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.724397 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.725073 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.725249 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.725313 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.726431 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:21.726432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:21.726432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:21.726436 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:21.727606 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.727894 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.727895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.727895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.727947 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:21.728098 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:21.728098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:21.728099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:21.728217 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:21.728286 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.728389 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.728444 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:50:21.729938 9260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:21.731563 9260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:21.737527 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.733839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.5.15.131:33333)
0926 14:50:21.733990 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.733994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.734289 (+ 295us) server_negotiation.cc:407] Connection header received
0926 14:50:21.734331 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.734333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.734347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.734434 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:21.734582 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.735593 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.736588 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.737078 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.737164 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.737221 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.737416 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.737454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260926 14:50:21.737519 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.733646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:33333 (local address 127.0.0.1:38898)
0926 14:50:21.734216 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.734226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.734273 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.734440 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.734441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.734443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:21.734501 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.734502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.735668 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.735669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.736502 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.736504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.737164 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.737165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.737250 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.737263 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.737386 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.737450 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260926 14:50:21.737979 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.733629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:37619 (local address 127.0.0.1:50960)
0926 14:50:21.733779 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.733830 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.733894 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.734131 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.734133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.734135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:21.734196 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.734198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.735619 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.735620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.736701 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.736702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.737594 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.737595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.737777 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.737782 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.737836 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.737892 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":150,"thread_start_us":51,"threads_started":1}
I20260926 14:50:21.738270 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.733802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.5.15.129:37619)
0926 14:50:21.733979 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.733984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.733994 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:21.734034 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.734036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.734051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.734129 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:21.734279 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.735524 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.736782 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.737467 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.737521 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.737566 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.738140 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.738180 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260926 14:50:21.738204 9650 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd31db6277841db87cc70ea6b64b786 (DEFAULT_TABLE table=test-table [id=a78066aaa6dc4f8789582d9823b235f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:21.738336 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.733804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.5.15.130:44861)
0926 14:50:21.734054 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.734057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.734067 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:21.734096 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.734098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.734109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.734202 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:21.734333 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.735233 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.736663 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.737373 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.737465 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.737521 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.738021 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.738249 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260926 14:50:21.737979 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.733629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:44861 (local address 127.0.0.1:48070)
0926 14:50:21.733993 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.734002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.734036 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.734200 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.734201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.734203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:21.734249 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.734250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.735323 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.735325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.736569 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.736571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.737555 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.737556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.737777 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.737783 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.737836 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.737893 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":14,"thread_start_us":58,"threads_started":1}
I20260926 14:50:21.738557 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd31db6277841db87cc70ea6b64b786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:21.738938 9384 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd31db6277841db87cc70ea6b64b786 (DEFAULT_TABLE table=test-table [id=a78066aaa6dc4f8789582d9823b235f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:21.739020 9517 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd31db6277841db87cc70ea6b64b786 (DEFAULT_TABLE table=test-table [id=a78066aaa6dc4f8789582d9823b235f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:21.739253 9384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd31db6277841db87cc70ea6b64b786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:21.739269 9517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd31db6277841db87cc70ea6b64b786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:21.741192 9730 tablet_bootstrap.cc:492] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Bootstrap starting.
I20260926 14:50:21.741994 9730 tablet_bootstrap.cc:654] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:21.742370 9732 tablet_bootstrap.cc:492] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Bootstrap starting.
I20260926 14:50:21.742856 9730 log.cc:824] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:21.742909 9731 tablet_bootstrap.cc:492] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: Bootstrap starting.
I20260926 14:50:21.743069 9732 tablet_bootstrap.cc:654] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:21.743499 9731 tablet_bootstrap.cc:654] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:21.743649 9732 log.cc:824] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:21.743755 9730 tablet_bootstrap.cc:492] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: No bootstrap required, opened a new log
I20260926 14:50:21.743821 9730 ts_tablet_manager.cc:1397] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:50:21.744094 9731 log.cc:824] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:21.744454 9732 tablet_bootstrap.cc:492] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: No bootstrap required, opened a new log
I20260926 14:50:21.744540 9732 ts_tablet_manager.cc:1397] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:50:21.744820 9731 tablet_bootstrap.cc:492] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: No bootstrap required, opened a new log
I20260926 14:50:21.744931 9731 ts_tablet_manager.cc:1397] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:21.745798 9730 raft_consensus.cc:348] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.745965 9730 raft_consensus.cc:374] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:21.746001 9730 raft_consensus.cc:729] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b45af04aee34f8f80cfb14affdd3b30, State: Initialized, Role: FOLLOWER
I20260926 14:50:21.746136 9730 consensus_queue.cc:260] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.746407 9730 ts_tablet_manager.cc:1428] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:21.746531 9716 heartbeater.cc:503] Master 127.5.15.190:41105 was elected leader, sending a full tablet report...
I20260926 14:50:21.746656 9732 raft_consensus.cc:348] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.746860 9732 raft_consensus.cc:374] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:21.746708 9731 raft_consensus.cc:348] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.746905 9732 raft_consensus.cc:729] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ed5486fdf2412196e25d8d3aebde1f, State: Initialized, Role: FOLLOWER
I20260926 14:50:21.746917 9731 raft_consensus.cc:374] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:21.746978 9731 raft_consensus.cc:729] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7420dd8bc543a6ab8725f61ae1b595, State: Initialized, Role: FOLLOWER
I20260926 14:50:21.747041 9732 consensus_queue.cc:260] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.747087 9731 consensus_queue.cc:260] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.747303 9732 ts_tablet_manager.cc:1428] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 14:50:21.747339 9450 heartbeater.cc:503] Master 127.5.15.190:41105 was elected leader, sending a full tablet report...
I20260926 14:50:21.747349 9731 ts_tablet_manager.cc:1428] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 14:50:21.747449 9583 heartbeater.cc:503] Master 127.5.15.190:41105 was elected leader, sending a full tablet report...
W20260926 14:50:21.806689 9584 tablet.cc:2369] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:21.881060 9738 raft_consensus.cc:482] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:21.881167 9738 raft_consensus.cc:504] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.881552 9738 leader_election.cc:288] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6ed5486fdf2412196e25d8d3aebde1f (127.5.15.130:44861), 1b45af04aee34f8f80cfb14affdd3b30 (127.5.15.131:33333)
I20260926 14:50:21.884415 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.881709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:44861 (local address 127.5.15.129:51561)
0926 14:50:21.881772 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.881785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.881836 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.882022 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.882022 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.882024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:21.882102 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.882103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.882789 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.882790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.883623 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.883625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.884159 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.884159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.884240 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.884243 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.884306 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.884342 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:50:21.884407 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.881806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:51561 (local address 127.5.15.130:44861)
0926 14:50:21.881872 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.881874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.881883 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:21.881934 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.881935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.881946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.882017 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:21.882188 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.882658 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.883713 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.884075 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.884159 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.884167 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.884310 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.884345 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:50:21.884662 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.881823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47915 (local address 127.5.15.131:33333)
0926 14:50:21.881874 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.881877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.881997 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:50:21.882075 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.882076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.882093 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.882172 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:21.882317 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.882838 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.883900 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.884298 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.884389 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.884398 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.884560 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.884602 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:21.884759 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd31db6277841db87cc70ea6b64b786" candidate_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" is_pre_election: true
I20260926 14:50:21.884908 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.881709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:33333 (local address 127.5.15.129:47915)
0926 14:50:21.881920 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.881935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:21.882009 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:21.882174 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:21.882175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:21.882176 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:21.882249 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.882251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.882964 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.882965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.883816 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:21.883819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.884392 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.884393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:21.884464 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.884467 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:21.884767 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 14:50:21.884811 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":52,"thread_start_us":72,"threads_started":1}
I20260926 14:50:21.884913 9537 raft_consensus.cc:2454] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7420dd8bc543a6ab8725f61ae1b595 in term 0.
I20260926 14:50:21.885166 9337 leader_election.cc:302] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b7420dd8bc543a6ab8725f61ae1b595, a6ed5486fdf2412196e25d8d3aebde1f; no voters:
I20260926 14:50:21.885254 9670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd31db6277841db87cc70ea6b64b786" candidate_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b45af04aee34f8f80cfb14affdd3b30" is_pre_election: true
I20260926 14:50:21.885370 9738 raft_consensus.cc:2781] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:21.885418 9670 raft_consensus.cc:2454] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7420dd8bc543a6ab8725f61ae1b595 in term 0.
I20260926 14:50:21.885473 9738 raft_consensus.cc:482] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:21.885504 9738 raft_consensus.cc:3037] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:21.886188 9738 raft_consensus.cc:504] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.886332 9738 leader_election.cc:288] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [CANDIDATE]: Term 1 election: Requested vote from peers a6ed5486fdf2412196e25d8d3aebde1f (127.5.15.130:44861), 1b45af04aee34f8f80cfb14affdd3b30 (127.5.15.131:33333)
I20260926 14:50:21.886458 9670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd31db6277841db87cc70ea6b64b786" candidate_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b45af04aee34f8f80cfb14affdd3b30"
I20260926 14:50:21.886520 9537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd31db6277841db87cc70ea6b64b786" candidate_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6ed5486fdf2412196e25d8d3aebde1f"
I20260926 14:50:21.886533 9670 raft_consensus.cc:3037] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:21.886618 9537 raft_consensus.cc:3037] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:21.887265 9670 raft_consensus.cc:2454] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7420dd8bc543a6ab8725f61ae1b595 in term 1.
I20260926 14:50:21.887434 9537 raft_consensus.cc:2454] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7420dd8bc543a6ab8725f61ae1b595 in term 1.
I20260926 14:50:21.887434 9340 leader_election.cc:302] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b45af04aee34f8f80cfb14affdd3b30, 8b7420dd8bc543a6ab8725f61ae1b595; no voters:
I20260926 14:50:21.887607 9738 raft_consensus.cc:2781] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:21.887787 9738 raft_consensus.cc:686] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7420dd8bc543a6ab8725f61ae1b595, State: Running, Role: LEADER
I20260926 14:50:21.887884 9738 consensus_queue.cc:237] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } }
I20260926 14:50:21.888712 9259 catalog_manager.cc:5495] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7420dd8bc543a6ab8725f61ae1b595 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b7420dd8bc543a6ab8725f61ae1b595" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 37619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 } health_report { overall_health: UNKNOWN } } }
W20260926 14:50:21.890184 9451 tablet.cc:2369] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:21.917718 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.915764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.5.15.129:37619)
0926 14:50:21.915818 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:21.915820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:21.915828 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:21.915859 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:21.915860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:21.915884 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:21.915950 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:21.916060 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.916486 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.917154 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:21.917317 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:21.917362 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:21.917481 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:21.917566 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:21.917622 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:21.917654 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 14:50:21.917668 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:21.919310 9537 raft_consensus.cc:1261] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 1 FOLLOWER]: Refusing update from remote peer 8b7420dd8bc543a6ab8725f61ae1b595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:21.919315 9670 raft_consensus.cc:1261] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 1 FOLLOWER]: Refusing update from remote peer 8b7420dd8bc543a6ab8725f61ae1b595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:21.919639 9738 consensus_queue.cc:1035] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ed5486fdf2412196e25d8d3aebde1f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 44861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:21.919795 9740 consensus_queue.cc:1035] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b45af04aee34f8f80cfb14affdd3b30" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 33333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:21.921792 9743 mvcc.cc:205] Tried to move back new op lower bound from 7333618572979302400 to 7333618572853579776. Current Snapshot: MvccSnapshot[applied={T|T < 7333618572979302400}]
I20260926 14:50:21.922498 9745 mvcc.cc:205] Tried to move back new op lower bound from 7333618572979302400 to 7333618572853579776. Current Snapshot: MvccSnapshot[applied={T|T < 7333618572979302400 or (T in {7333618572979302400})}]
I20260926 14:50:21.922837 9744 mvcc.cc:205] Tried to move back new op lower bound from 7333618572979302400 to 7333618572853579776. Current Snapshot: MvccSnapshot[applied={T|T < 7333618572979302400}]
I20260926 14:50:21.924099 9260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:21.924192 9260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:21.924893 9260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6129666498de4d4bb78eb731194db0ed: Sending DeleteTablet for 3 replicas of tablet 1dd31db6277841db87cc70ea6b64b786
I20260926 14:50:21.925189 9517 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd31db6277841db87cc70ea6b64b786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:21 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:48070
I20260926 14:50:21.925197 9384 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd31db6277841db87cc70ea6b64b786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:21 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:50960
I20260926 14:50:21.925329 9650 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd31db6277841db87cc70ea6b64b786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:21 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:38898
I20260926 14:50:21.925447 9753 tablet_replica.cc:330] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595: stopping tablet replica
I20260926 14:50:21.925580 9753 raft_consensus.cc:2229] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:21.925618 9752 tablet_replica.cc:330] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: stopping tablet replica
I20260926 14:50:21.925715 9752 raft_consensus.cc:2229] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:21.925788 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9322
I20260926 14:50:21.925808 9752 raft_consensus.cc:2258] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:21.925897 9753 raft_consensus.cc:2258] T 1dd31db6277841db87cc70ea6b64b786 P 8b7420dd8bc543a6ab8725f61ae1b595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:21.929673 9752 ts_tablet_manager.cc:1905] T 1dd31db6277841db87cc70ea6b64b786 P a6ed5486fdf2412196e25d8d3aebde1f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:50:21.930344 9245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:37619 (error 108)
I20260926 14:50:21.925524 9754 tablet_replica.cc:330] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: stopping tablet replica
I20260926 14:50:21.930785 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9455
I20260926 14:50:21.930826 9754 raft_consensus.cc:2229] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:21.930866 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.930679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:37619 (local address 127.0.0.1:50982)
0926 14:50:21.930736 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.930835 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:37619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:50:21.930938 9754 raft_consensus.cc:2258] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:50:21.931468 9245 catalog_manager.cc:4584] TS 8b7420dd8bc543a6ab8725f61ae1b595 (127.5.15.129:37619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1dd31db6277841db87cc70ea6b64b786: Network error: Client connection negotiation failed: client connection to 127.5.15.129:37619: connect: Connection refused (error 111)
I20260926 14:50:21.932031 9754 ts_tablet_manager.cc:1905] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:21.933511 9754 ts_tablet_manager.cc:1918] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:21.933593 9754 log.cc:1196] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/1dd31db6277841db87cc70ea6b64b786
I20260926 14:50:21.933841 9754 ts_tablet_manager.cc:1939] T 1dd31db6277841db87cc70ea6b64b786 P 1b45af04aee34f8f80cfb14affdd3b30: Deleting consensus metadata
I20260926 14:50:21.934237 9247 catalog_manager.cc:4841] TS 1b45af04aee34f8f80cfb14affdd3b30 (127.5.15.131:33333): tablet 1dd31db6277841db87cc70ea6b64b786 (table test-table [id=a78066aaa6dc4f8789582d9823b235f2]) successfully deleted
I20260926 14:50:21.935544 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:21.935419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:44861 (local address 127.0.0.1:48078)
0926 14:50:21.935483 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:21.935515 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.130:44861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:50:21.935532 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9588
I20260926 14:50:21.940006 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9229
2026-09-26T14:50:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1981 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T14:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:21Z Disabled control of system clock
I20260926 14:50:21.967213 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:46449
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:50:22.045401 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.045598 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.045619 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.046854 9759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:22.046893 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.046906 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.046919 9759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:22.046931 9759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:22.048286 9759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:46449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:22.048486 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.048732 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.053251 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.053290 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.053277 9767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.053440 9759 server_base.cc:1034] running on GCE node
I20260926 14:50:22.053689 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.053956 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.055088 9759 hybrid_clock.cc:648] HybridClock initialized: now 1790434222055077 us; error 26 us; skew 500 ppm
I20260926 14:50:22.056339 9759 webserver.cc:460] Webserver started at http://127.5.15.190:36985/ using document root <none> and password file <none>
I20260926 14:50:22.056514 9759 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.056551 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.056636 9759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.057559 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "70df442595c34c0a94a87d0015462b88"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "7018a47e36832d37d356e166c8012aa6"
server_key_iv: "4afe419c3d5ae83e0e222bc8244b9efe"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.057832 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "70df442595c34c0a94a87d0015462b88"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "7018a47e36832d37d356e166c8012aa6"
server_key_iv: "4afe419c3d5ae83e0e222bc8244b9efe"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.059067 9759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:22.059744 9773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.059953 9759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:22.060029 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "70df442595c34c0a94a87d0015462b88"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "7018a47e36832d37d356e166c8012aa6"
server_key_iv: "4afe419c3d5ae83e0e222bc8244b9efe"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.060076 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.063298 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.063508 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.063596 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.067185 9759 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:46449
I20260926 14:50:22.067225 9825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:46449 every 8 connection(s)
I20260926 14:50:22.067533 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:22.068140 9826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:22.070577 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88: Bootstrap starting.
I20260926 14:50:22.070909 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9759
I20260926 14:50:22.070976 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:22.071133 5182 external_mini_cluster.cc:1428] Setting key 5a328e541ca9071df97ccb4ce22b008c
I20260926 14:50:22.071259 9826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:22.071611 9826 log.cc:824] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:22.072230 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88: No bootstrap required, opened a new log
I20260926 14:50:22.073395 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.071567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.5.15.190:46449)
0926 14:50:22.071771 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.071780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.071817 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:50:22.072190 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.072195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.072228 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.072375 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.073000 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.073003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.073003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.073008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.073060 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.073069 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.073162 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.073197 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.073247 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260926 14:50:22.074015 9826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70df442595c34c0a94a87d0015462b88" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46449 } }
I20260926 14:50:22.074168 9826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:22.074211 9826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70df442595c34c0a94a87d0015462b88, State: Initialized, Role: FOLLOWER
I20260926 14:50:22.074318 9826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70df442595c34c0a94a87d0015462b88" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46449 } }
I20260926 14:50:22.074424 9826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:22.074467 9826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:22.074512 9826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:22.075366 9826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70df442595c34c0a94a87d0015462b88" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46449 } }
I20260926 14:50:22.075497 9826 leader_election.cc:302] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70df442595c34c0a94a87d0015462b88; no voters:
I20260926 14:50:22.075646 9826 leader_election.cc:288] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:22.075670 9831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:22.075834 9831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [term 1 LEADER]: Becoming Leader. State: Replica: 70df442595c34c0a94a87d0015462b88, State: Running, Role: LEADER
I20260926 14:50:22.075914 9826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:22.075933 9831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70df442595c34c0a94a87d0015462b88" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46449 } }
I20260926 14:50:22.076267 9833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70df442595c34c0a94a87d0015462b88. Latest consensus state: current_term: 1 leader_uuid: "70df442595c34c0a94a87d0015462b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70df442595c34c0a94a87d0015462b88" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46449 } } }
I20260926 14:50:22.076269 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70df442595c34c0a94a87d0015462b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70df442595c34c0a94a87d0015462b88" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46449 } } }
I20260926 14:50:22.076469 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:22.076663 9833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:22.076787 9840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:22.077247 9840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:22.078639 9840 catalog_manager.cc:1347] Generated new cluster ID: 5bcb4d07877146d587368ef342789ad4
I20260926 14:50:22.078694 9840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:22.111430 9840 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:22.111598 9840 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:22.117261 9840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88: Generated new TSK 0
I20260926 14:50:22.117439 9840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:22.128083 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46449
--builtin_ntp_servers=127.5.15.148:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:50:22.208839 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.209018 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.209038 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.210073 9850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.210105 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.210129 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:22.211376 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:22.211695 9850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.211932 9850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.216459 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.216557 9858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.216610 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.216840 9850 server_base.cc:1034] running on GCE node
I20260926 14:50:22.217056 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.217324 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.218479 9850 hybrid_clock.cc:648] HybridClock initialized: now 1790434222218467 us; error 42 us; skew 500 ppm
I20260926 14:50:22.219664 9850 webserver.cc:460] Webserver started at http://127.5.15.129:40865/ using document root <none> and password file <none>
I20260926 14:50:22.219901 9850 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.219966 9850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.220091 9850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.221076 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "445244ecb976453ba268ce6da12099dc"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "dc7d70273643dba06f69e5e810c56644"
server_key_iv: "6b38de7dfd46a7cb715d005311b005cb"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.221437 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "445244ecb976453ba268ce6da12099dc"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "dc7d70273643dba06f69e5e810c56644"
server_key_iv: "6b38de7dfd46a7cb715d005311b005cb"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.222661 9850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:22.223358 9864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.223537 9850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.223625 9850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "445244ecb976453ba268ce6da12099dc"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "dc7d70273643dba06f69e5e810c56644"
server_key_iv: "6b38de7dfd46a7cb715d005311b005cb"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.223699 9850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.226842 9850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.227082 9850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.227205 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.227406 9850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:22.227671 9850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:22.227720 9850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.227756 9850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:22.227783 9850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.233086 9850 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:45577
I20260926 14:50:22.233140 9977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:45577 every 8 connection(s)
I20260926 14:50:22.233407 9850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:22.235930 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.234038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:60139 (local address 127.5.15.190:46449)
0926 14:50:22.234095 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.234097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.234310 (+ 213us) server_negotiation.cc:407] Connection header received
0926 14:50:22.234677 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.234678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.234682 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.234732 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.235758 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.235760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.235760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.235764 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.235775 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.235779 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.235824 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.235869 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.235881 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:50:22.236127 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.234029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:46449 (local address 127.5.15.129:60139)
0926 14:50:22.234198 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.234233 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.234603 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.234840 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.234843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.234851 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.235616 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.235617 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.235639 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.235663 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.235822 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.235824 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.235827 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.235876 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.235918 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 14:50:22.236600 9978 heartbeater.cc:347] Connected to a master server at 127.5.15.190:46449
I20260926 14:50:22.236689 9978 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:22.236833 9978 heartbeater.cc:511] Master 127.5.15.190:46449 requested a full tablet report, sending...
I20260926 14:50:22.237295 9790 ts_manager.cc:194] Registered new tserver with Master: 445244ecb976453ba268ce6da12099dc (127.5.15.129:45577)
I20260926 14:50:22.243156 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9850
I20260926 14:50:22.243233 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:22.243352 5182 external_mini_cluster.cc:1428] Setting key f6575a0d1c69f18a4543cfc23aef4c6e
I20260926 14:50:22.244113 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46449
--builtin_ntp_servers=127.5.15.148:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:50:22.320111 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.320286 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.320309 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.321404 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.321449 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.321501 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:22.322732 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:22.323004 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.323261 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.329393 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.329643 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.329409 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.329979 9981 server_base.cc:1034] running on GCE node
I20260926 14:50:22.330164 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.330394 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.331561 9981 hybrid_clock.cc:648] HybridClock initialized: now 1790434222331545 us; error 52 us; skew 500 ppm
I20260926 14:50:22.332710 9981 webserver.cc:460] Webserver started at http://127.5.15.130:43265/ using document root <none> and password file <none>
I20260926 14:50:22.332957 9981 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.333024 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.333186 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.334137 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "a16547af7e424727a3b30c0756cc5c31"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "d77f15a4f8bd6e4c7812ed281f15cca7"
server_key_iv: "7db92763b893a46972d1541e235a76e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.334486 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "a16547af7e424727a3b30c0756cc5c31"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "d77f15a4f8bd6e4c7812ed281f15cca7"
server_key_iv: "7db92763b893a46972d1541e235a76e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.335815 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:22.336493 9995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.336665 9981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:22.336757 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "a16547af7e424727a3b30c0756cc5c31"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "d77f15a4f8bd6e4c7812ed281f15cca7"
server_key_iv: "7db92763b893a46972d1541e235a76e8"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.336841 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.340027 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.340250 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.340402 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.340590 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:22.340865 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:22.340916 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.340952 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:22.340979 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.346318 9981 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:42661
I20260926 14:50:22.346354 10108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:42661 every 8 connection(s)
I20260926 14:50:22.346642 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:22.347839 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 9981
I20260926 14:50:22.347939 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:22.348095 5182 external_mini_cluster.cc:1428] Setting key fd553f8ed29744665238c702353fe68d
I20260926 14:50:22.348990 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46449
--builtin_ntp_servers=127.5.15.148:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 14:50:22.350833 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.347328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:44715 (local address 127.5.15.190:46449)
0926 14:50:22.347408 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.347410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.347622 (+ 212us) server_negotiation.cc:407] Connection header received
0926 14:50:22.348116 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.348117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.348124 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.348171 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.350556 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.350558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.350558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.350563 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.350578 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.350581 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.350628 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.350709 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.350740 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:50:22.350996 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.347321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:46449 (local address 127.5.15.130:44715)
0926 14:50:22.347521 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.347549 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.348007 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.348256 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.348259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.348270 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.350421 (+ 2151us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.350421 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.350444 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.350468 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.350626 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.350629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.350633 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.350736 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.350807 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260926 14:50:22.351507 10109 heartbeater.cc:347] Connected to a master server at 127.5.15.190:46449
I20260926 14:50:22.351596 10109 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:22.351734 10109 heartbeater.cc:511] Master 127.5.15.190:46449 requested a full tablet report, sending...
I20260926 14:50:22.352047 9790 ts_manager.cc:194] Registered new tserver with Master: a16547af7e424727a3b30c0756cc5c31 (127.5.15.130:42661)
W20260926 14:50:22.426940 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.427152 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.427184 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.428288 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.428351 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.428393 10112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:22.429672 10112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:22.429924 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.430188 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.434978 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.434988 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.434988 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.435102 10112 server_base.cc:1034] running on GCE node
I20260926 14:50:22.435444 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.435688 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.436829 10112 hybrid_clock.cc:648] HybridClock initialized: now 1790434222436813 us; error 41 us; skew 500 ppm
I20260926 14:50:22.438122 10112 webserver.cc:460] Webserver started at http://127.5.15.131:35695/ using document root <none> and password file <none>
I20260926 14:50:22.438376 10112 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.438423 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.438566 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.439706 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "bb953b00ebe34944afb239c5208c2723"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "c76a6e3997799449ff9ca4548dc75913"
server_key_iv: "572a2eeecc543bc0cfb3d9d52924e935"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.440054 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "bb953b00ebe34944afb239c5208c2723"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "c76a6e3997799449ff9ca4548dc75913"
server_key_iv: "572a2eeecc543bc0cfb3d9d52924e935"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.441296 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:22.442004 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.442166 10112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:22.442240 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "bb953b00ebe34944afb239c5208c2723"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "c76a6e3997799449ff9ca4548dc75913"
server_key_iv: "572a2eeecc543bc0cfb3d9d52924e935"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.442289 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.445369 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.445554 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.445639 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.445783 10112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:22.446024 10112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:22.446053 10112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.446071 10112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:22.446084 10112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.451295 10112 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:44435
I20260926 14:50:22.451360 10239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:44435 every 8 connection(s)
I20260926 14:50:22.451649 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:22.453814 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10112
I20260926 14:50:22.453922 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:22.454131 5182 external_mini_cluster.cc:1428] Setting key ed404413bd53be63d5b68e7ea7ed7339
I20260926 14:50:22.454624 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.452320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:36177 (local address 127.5.15.190:46449)
0926 14:50:22.452380 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.452382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.452590 (+ 208us) server_negotiation.cc:407] Connection header received
0926 14:50:22.452987 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.452988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.452993 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.453054 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.454399 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.454401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.454401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.454406 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.454422 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.454426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.454471 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.454543 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.454557 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:50:22.454902 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.452299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:46449 (local address 127.5.15.131:36177)
0926 14:50:22.452471 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.452508 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.452912 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.453102 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.453104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.453112 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.454222 (+ 1110us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.454222 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.454259 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.454291 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.454480 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.454485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.454488 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.454555 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.454610 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 14:50:22.455263 10240 heartbeater.cc:347] Connected to a master server at 127.5.15.190:46449
I20260926 14:50:22.455339 10240 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:22.455436 10240 heartbeater.cc:511] Master 127.5.15.190:46449 requested a full tablet report, sending...
I20260926 14:50:22.455724 9790 ts_manager.cc:194] Registered new tserver with Master: bb953b00ebe34944afb239c5208c2723 (127.5.15.131:44435)
I20260926 14:50:22.465780 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:22.468052 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.467056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.5.15.190:46449)
0926 14:50:22.467098 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.467100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.467205 (+ 105us) server_negotiation.cc:407] Connection header received
0926 14:50:22.467263 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.467264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.467268 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.467324 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.467910 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.467910 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.467910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.467913 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.467922 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.467925 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.467962 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.467997 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.468007 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:22.469445 9790 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:22.470945 9790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:22.475128 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.473020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:44435 (local address 127.0.0.1:47238)
0926 14:50:22.473558 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.473566 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.473612 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.474031 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.474032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.474033 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.474688 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.474688 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.474694 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.474704 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.474989 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.474990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.474991 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.475047 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.475062 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":92,"threads_started":1}
I20260926 14:50:22.475158 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.473171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.5.15.130:42661)
0926 14:50:22.473314 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.473319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.473490 (+ 171us) server_negotiation.cc:407] Connection header received
0926 14:50:22.473557 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.473564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.473578 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.473654 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.474776 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.474784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.474784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.474789 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.474836 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.474861 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.474926 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.475065 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.475082 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260926 14:50:22.475342 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.472997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:45577 (local address 127.0.0.1:47040)
0926 14:50:22.473349 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.473401 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.473472 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.474015 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.474022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.474027 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.474985 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.474986 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.474991 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.475002 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.475248 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.475249 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.475250 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.475285 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.475299 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":67,"threads_started":1}
I20260926 14:50:22.475435 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.472997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:42661 (local address 127.0.0.1:40758)
0926 14:50:22.473375 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.473405 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.473488 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.474014 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.474019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.474025 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.474629 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.474629 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.474646 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.474670 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.474946 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.474953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.474956 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.475343 (+ 387us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.475358 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":3,"thread_start_us":78,"threads_started":1}
I20260926 14:50:22.475586 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.473169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.5.15.129:45577)
0926 14:50:22.473399 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.473403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.473530 (+ 127us) server_negotiation.cc:407] Connection header received
0926 14:50:22.473559 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.473561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.473571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.473646 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.475084 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.475091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.475091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.475095 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.475129 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.475158 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.475432 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.475461 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.475480 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260926 14:50:22.475157 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.473241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.5.15.131:44435)
0926 14:50:22.473471 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.473474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.473642 (+ 168us) server_negotiation.cc:407] Connection header received
0926 14:50:22.473667 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.473727 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.473738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.473803 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.474824 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.474831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.474831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.474842 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.474871 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.474895 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.474983 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.475052 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.475068 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":68,"threads_started":1}
I20260926 14:50:22.476148 10043 tablet_service.cc:1467] Processing CreateTablet for tablet bd905e0841184d5c9521f2b75c379e2b (DEFAULT_TABLE table=test-table [id=384b2bb5a4d84ea28507b1423e4aff45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:22.476148 10174 tablet_service.cc:1467] Processing CreateTablet for tablet bd905e0841184d5c9521f2b75c379e2b (DEFAULT_TABLE table=test-table [id=384b2bb5a4d84ea28507b1423e4aff45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:22.476320 9912 tablet_service.cc:1467] Processing CreateTablet for tablet bd905e0841184d5c9521f2b75c379e2b (DEFAULT_TABLE table=test-table [id=384b2bb5a4d84ea28507b1423e4aff45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:22.476517 10043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd905e0841184d5c9521f2b75c379e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:22.476517 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd905e0841184d5c9521f2b75c379e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:22.476686 9912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd905e0841184d5c9521f2b75c379e2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:22.479216 10254 tablet_bootstrap.cc:492] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: Bootstrap starting.
I20260926 14:50:22.479698 10256 tablet_bootstrap.cc:492] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Bootstrap starting.
I20260926 14:50:22.480039 10254 tablet_bootstrap.cc:654] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:22.480330 10256 tablet_bootstrap.cc:654] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:22.480366 10254 log.cc:824] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:22.480641 10256 log.cc:824] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:22.480960 10255 tablet_bootstrap.cc:492] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Bootstrap starting.
I20260926 14:50:22.481128 10254 tablet_bootstrap.cc:492] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: No bootstrap required, opened a new log
I20260926 14:50:22.481204 10254 ts_tablet_manager.cc:1397] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:22.481374 10256 tablet_bootstrap.cc:492] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: No bootstrap required, opened a new log
I20260926 14:50:22.481462 10256 ts_tablet_manager.cc:1397] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:22.481559 10255 tablet_bootstrap.cc:654] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:22.481864 10255 log.cc:824] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:22.483413 10255 tablet_bootstrap.cc:492] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: No bootstrap required, opened a new log
I20260926 14:50:22.483482 10255 ts_tablet_manager.cc:1397] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:50:22.483606 10254 raft_consensus.cc:348] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.483758 10254 raft_consensus.cc:374] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:22.483803 10254 raft_consensus.cc:729] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445244ecb976453ba268ce6da12099dc, State: Initialized, Role: FOLLOWER
I20260926 14:50:22.483698 10256 raft_consensus.cc:348] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.483843 10256 raft_consensus.cc:374] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:22.483875 10256 raft_consensus.cc:729] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16547af7e424727a3b30c0756cc5c31, State: Initialized, Role: FOLLOWER
I20260926 14:50:22.483932 10254 consensus_queue.cc:260] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.483983 10256 consensus_queue.cc:260] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.484215 10256 ts_tablet_manager.cc:1428] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:22.484223 9978 heartbeater.cc:503] Master 127.5.15.190:46449 was elected leader, sending a full tablet report...
I20260926 14:50:22.484326 10109 heartbeater.cc:503] Master 127.5.15.190:46449 was elected leader, sending a full tablet report...
I20260926 14:50:22.484457 10254 ts_tablet_manager.cc:1428] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:22.485919 10255 raft_consensus.cc:348] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.486038 10255 raft_consensus.cc:374] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:22.486059 10255 raft_consensus.cc:729] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb953b00ebe34944afb239c5208c2723, State: Initialized, Role: FOLLOWER
I20260926 14:50:22.486188 10255 consensus_queue.cc:260] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.486429 10255 ts_tablet_manager.cc:1428] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:22.486477 10240 heartbeater.cc:503] Master 127.5.15.190:46449 was elected leader, sending a full tablet report...
W20260926 14:50:22.597193 10110 tablet.cc:2369] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:22.679242 10262 raft_consensus.cc:482] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:22.679360 10262 raft_consensus.cc:504] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.679765 10262 leader_election.cc:288] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445244ecb976453ba268ce6da12099dc (127.5.15.129:45577), a16547af7e424727a3b30c0756cc5c31 (127.5.15.130:42661)
I20260926 14:50:22.681058 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.679935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:45577 (local address 127.5.15.131:41599)
0926 14:50:22.680025 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.680039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.680079 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.680255 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.680256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.680258 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.680687 (+ 429us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.680687 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.680694 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.680704 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.680913 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.680914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.680914 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.680967 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.680979 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 14:50:22.681084 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.680007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:41599 (local address 127.5.15.129:45577)
0926 14:50:22.680057 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.680059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.680073 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:50:22.680142 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.680143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.680149 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.680239 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.680818 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.680820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.680821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.680825 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.680843 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.680848 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.680903 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.680984 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.680998 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 14:50:22.681206 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.680054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:50215 (local address 127.5.15.130:42661)
0926 14:50:22.680115 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.680118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.680197 (+ 79us) server_negotiation.cc:407] Connection header received
0926 14:50:22.680238 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.680239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.680243 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.680300 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.680969 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.680970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.680970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.680975 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.680990 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.680995 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.681050 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.681117 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.681130 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:22.681212 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.679966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:42661 (local address 127.5.15.131:50215)
0926 14:50:22.680127 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.680137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.680190 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.680306 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.680307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.680309 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:22.680854 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:22.680855 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:22.680864 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:22.680871 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:22.681055 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:22.681055 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:22.681056 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:50:22.681116 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:50:22.681128 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260926 14:50:22.681414 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd905e0841184d5c9521f2b75c379e2b" candidate_uuid: "bb953b00ebe34944afb239c5208c2723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445244ecb976453ba268ce6da12099dc" is_pre_election: true
I20260926 14:50:22.681559 9932 raft_consensus.cc:2454] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb953b00ebe34944afb239c5208c2723 in term 0.
I20260926 14:50:22.681754 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd905e0841184d5c9521f2b75c379e2b" candidate_uuid: "bb953b00ebe34944afb239c5208c2723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16547af7e424727a3b30c0756cc5c31" is_pre_election: true
I20260926 14:50:22.681819 10128 leader_election.cc:302] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445244ecb976453ba268ce6da12099dc, bb953b00ebe34944afb239c5208c2723; no voters:
I20260926 14:50:22.681887 10063 raft_consensus.cc:2454] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb953b00ebe34944afb239c5208c2723 in term 0.
I20260926 14:50:22.681924 10262 raft_consensus.cc:2781] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:22.681967 10262 raft_consensus.cc:482] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:22.681985 10262 raft_consensus.cc:3037] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:22.682688 10262 raft_consensus.cc:504] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.682859 10262 leader_election.cc:288] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [CANDIDATE]: Term 1 election: Requested vote from peers 445244ecb976453ba268ce6da12099dc (127.5.15.129:45577), a16547af7e424727a3b30c0756cc5c31 (127.5.15.130:42661)
I20260926 14:50:22.683014 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd905e0841184d5c9521f2b75c379e2b" candidate_uuid: "bb953b00ebe34944afb239c5208c2723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445244ecb976453ba268ce6da12099dc"
I20260926 14:50:22.683017 10063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd905e0841184d5c9521f2b75c379e2b" candidate_uuid: "bb953b00ebe34944afb239c5208c2723" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16547af7e424727a3b30c0756cc5c31"
I20260926 14:50:22.683106 9932 raft_consensus.cc:3037] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:22.683103 10063 raft_consensus.cc:3037] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:22.683768 9932 raft_consensus.cc:2454] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb953b00ebe34944afb239c5208c2723 in term 1.
I20260926 14:50:22.683835 10063 raft_consensus.cc:2454] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb953b00ebe34944afb239c5208c2723 in term 1.
I20260926 14:50:22.684000 10128 leader_election.cc:302] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445244ecb976453ba268ce6da12099dc, bb953b00ebe34944afb239c5208c2723; no voters:
I20260926 14:50:22.684085 10262 raft_consensus.cc:2781] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:22.684265 10262 raft_consensus.cc:686] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 1 LEADER]: Becoming Leader. State: Replica: bb953b00ebe34944afb239c5208c2723, State: Running, Role: LEADER
I20260926 14:50:22.684351 10262 consensus_queue.cc:237] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } }
I20260926 14:50:22.685088 9789 catalog_manager.cc:5495] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb953b00ebe34944afb239c5208c2723 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "bb953b00ebe34944afb239c5208c2723" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb953b00ebe34944afb239c5208c2723" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44435 } health_report { overall_health: HEALTHY } } }
W20260926 14:50:22.702134 10241 tablet.cc:2369] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:22.704874 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.704092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.5.15.131:44435)
0926 14:50:22.704136 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.704137 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.704144 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:22.704182 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.704183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.704192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.704241 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.704719 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.704720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.704720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.704724 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.704736 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.704740 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.704787 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.704822 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.704832 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:22.706435 10063 raft_consensus.cc:1261] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 1 FOLLOWER]: Refusing update from remote peer bb953b00ebe34944afb239c5208c2723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:22.706490 9932 raft_consensus.cc:1261] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 1 FOLLOWER]: Refusing update from remote peer bb953b00ebe34944afb239c5208c2723: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:22.706746 10262 consensus_queue.cc:1035] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16547af7e424727a3b30c0756cc5c31" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:22.706875 10264 consensus_queue.cc:1035] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445244ecb976453ba268ce6da12099dc" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 45577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:22.709095 10267 mvcc.cc:205] Tried to move back new op lower bound from 7333618576203341824 to 7333618576115662848. Current Snapshot: MvccSnapshot[applied={T|T < 7333618576203341824}]
I20260926 14:50:22.709389 10268 mvcc.cc:205] Tried to move back new op lower bound from 7333618576203341824 to 7333618576115662848. Current Snapshot: MvccSnapshot[applied={T|T < 7333618576203341824 or (T in {7333618576203341824})}]
I20260926 14:50:22.710306 10269 mvcc.cc:205] Tried to move back new op lower bound from 7333618576203341824 to 7333618576115662848. Current Snapshot: MvccSnapshot[applied={T|T < 7333618576203341824 or (T in {7333618576203341824})}]
I20260926 14:50:22.711019 9788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:22.711117 9788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:22.711799 9788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70df442595c34c0a94a87d0015462b88: Sending DeleteTablet for 3 replicas of tablet bd905e0841184d5c9521f2b75c379e2b
I20260926 14:50:22.712131 9912 tablet_service.cc:1514] Processing DeleteTablet for tablet bd905e0841184d5c9521f2b75c379e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:22 UTC) from {username='slave'} at 127.0.0.1:47040
I20260926 14:50:22.712152 10043 tablet_service.cc:1514] Processing DeleteTablet for tablet bd905e0841184d5c9521f2b75c379e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:22 UTC) from {username='slave'} at 127.0.0.1:40758
I20260926 14:50:22.712335 10174 tablet_service.cc:1514] Processing DeleteTablet for tablet bd905e0841184d5c9521f2b75c379e2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:22 UTC) from {username='slave'} at 127.0.0.1:47238
I20260926 14:50:22.712383 10277 tablet_replica.cc:330] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc: stopping tablet replica
I20260926 14:50:22.712368 10278 tablet_replica.cc:330] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: stopping tablet replica
I20260926 14:50:22.712469 10277 raft_consensus.cc:2229] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:22.712567 10277 raft_consensus.cc:2258] T bd905e0841184d5c9521f2b75c379e2b P 445244ecb976453ba268ce6da12099dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:22.712576 10278 raft_consensus.cc:2229] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:22.712711 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9850
I20260926 14:50:22.712743 10278 raft_consensus.cc:2258] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:22.712958 10279 tablet_replica.cc:330] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: stopping tablet replica
I20260926 14:50:22.713068 10279 raft_consensus.cc:2229] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:22.713169 10279 raft_consensus.cc:2258] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:22.713564 10279 ts_tablet_manager.cc:1905] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:22.713564 10278 ts_tablet_manager.cc:1905] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:22.715198 10278 ts_tablet_manager.cc:1918] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:22.715198 10279 ts_tablet_manager.cc:1918] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:22.715270 10278 log.cc:1196] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/bd905e0841184d5c9521f2b75c379e2b
I20260926 14:50:22.715366 10279 log.cc:1196] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/bd905e0841184d5c9521f2b75c379e2b
I20260926 14:50:22.715544 10278 ts_tablet_manager.cc:1939] T bd905e0841184d5c9521f2b75c379e2b P a16547af7e424727a3b30c0756cc5c31: Deleting consensus metadata
I20260926 14:50:22.715740 10279 ts_tablet_manager.cc:1939] T bd905e0841184d5c9521f2b75c379e2b P bb953b00ebe34944afb239c5208c2723: Deleting consensus metadata
I20260926 14:50:22.716658 9774 catalog_manager.cc:4841] TS a16547af7e424727a3b30c0756cc5c31 (127.5.15.130:42661): tablet bd905e0841184d5c9521f2b75c379e2b (table test-table [id=384b2bb5a4d84ea28507b1423e4aff45]) successfully deleted
I20260926 14:50:22.716856 9776 catalog_manager.cc:4841] TS bb953b00ebe34944afb239c5208c2723 (127.5.15.131:44435): tablet bd905e0841184d5c9521f2b75c379e2b (table test-table [id=384b2bb5a4d84ea28507b1423e4aff45]) successfully deleted
W20260926 14:50:22.720810 9775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:45577 (error 108)
I20260926 14:50:22.721263 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.721138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:45577 (local address 127.0.0.1:47042)
0926 14:50:22.721188 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.721227 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:50:22.721304 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9981
W20260926 14:50:22.723134 9775 catalog_manager.cc:4584] TS 445244ecb976453ba268ce6da12099dc (127.5.15.129:45577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd905e0841184d5c9521f2b75c379e2b: Network error: Client connection negotiation failed: client connection to 127.5.15.129:45577: connect: Connection refused (error 111)
I20260926 14:50:22.726284 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10112
I20260926 14:50:22.730808 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 9759
2026-09-26T14:50:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (789 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 14:50:22.749848 5182 mini_oidc.cc:152] Starting JWKS server
I20260926 14:50:22.751142 5182 webserver.cc:460] Webserver started at https://127.0.0.1:45873/ using document root <none> and password file <none>
I20260926 14:50:22.751216 5182 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:50:22.751416 5182 webserver.cc:460] Webserver started at http://127.0.0.1:45859/ using document root <none> and password file <none>
2026-09-26T14:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:22Z Disabled control of system clock
I20260926 14:50:22.758536 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40297
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:43009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:40297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/testchainca.cert with env {}
W20260926 14:50:22.837456 10286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:22.837636 10286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:22.837656 10286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:22.838891 10286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:22.838929 10286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:22.838941 10286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:22.838979 10286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:22.839007 10286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:22.839025 10286 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:22.839046 10286 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45873/jwks.json
I20260926 14:50:22.840538 10286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:43009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:40297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:22.840781 10286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:22.840992 10286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:22.845649 10291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.845657 10294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:22.845660 10292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:22.845849 10286 server_base.cc:1034] running on GCE node
I20260926 14:50:22.846083 10286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:22.846390 10286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:22.847543 10286 hybrid_clock.cc:648] HybridClock initialized: now 1790434222847535 us; error 29 us; skew 500 ppm
I20260926 14:50:22.848753 10286 webserver.cc:460] Webserver started at http://127.5.15.190:44891/ using document root <none> and password file <none>
I20260926 14:50:22.848997 10286 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:22.849042 10286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:22.849207 10286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:22.850159 10286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "4f678044d4624ae8af21a53bac47b29d"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "444518fca40d7effa753da5342fdd402"
server_key_iv: "86b76895540bfa5e5ae9c567d082d56a"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.850507 10286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "4f678044d4624ae8af21a53bac47b29d"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "444518fca40d7effa753da5342fdd402"
server_key_iv: "86b76895540bfa5e5ae9c567d082d56a"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.851760 10286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:22.852449 10300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:22.852617 10286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:22.852679 10286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "4f678044d4624ae8af21a53bac47b29d"
format_stamp: "Formatted at 2026-09-26 14:50:22 on dist-test-slave-s4p1"
server_key: "444518fca40d7effa753da5342fdd402"
server_key_iv: "86b76895540bfa5e5ae9c567d082d56a"
server_key_version: "encryptionkey@0"
I20260926 14:50:22.852725 10286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:22.865743 10286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:22.865972 10286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:22.866063 10286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:22.869815 10286 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:40297
I20260926 14:50:22.869843 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:40297 every 8 connection(s)
I20260926 14:50:22.870167 10286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:22.871147 10353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:22.872202 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10286
I20260926 14:50:22.872314 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:22.872488 5182 external_mini_cluster.cc:1428] Setting key 6e6f32d68e2754d58d79f07968d7fe28
I20260926 14:50:22.873647 10353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d: Bootstrap starting.
I20260926 14:50:22.874214 10353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:22.874495 10353 log.cc:824] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:22.875174 10353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d: No bootstrap required, opened a new log
I20260926 14:50:22.876857 10353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f678044d4624ae8af21a53bac47b29d" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40297 } }
I20260926 14:50:22.877003 10353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:22.877049 10353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f678044d4624ae8af21a53bac47b29d, State: Initialized, Role: FOLLOWER
I20260926 14:50:22.877190 10353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f678044d4624ae8af21a53bac47b29d" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40297 } }
I20260926 14:50:22.877281 10353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:22.877326 10353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:22.877375 10353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:22.877782 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.872973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.5.15.190:40297)
0926 14:50:22.873174 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.873180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.873203 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:50:22.873575 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.873581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.873646 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.873796 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:22.873982 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.875061 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.875902 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.876309 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.876442 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.877075 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:22.877079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:22.877079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:22.877268 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:22.877299 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:22.877332 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:22.877423 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:22.877506 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:50:22.877584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260926 14:50:22.878022 10353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f678044d4624ae8af21a53bac47b29d" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40297 } }
I20260926 14:50:22.878134 10353 leader_election.cc:302] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f678044d4624ae8af21a53bac47b29d; no voters:
I20260926 14:50:22.878293 10353 leader_election.cc:288] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:22.878345 10358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:22.878497 10358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [term 1 LEADER]: Becoming Leader. State: Replica: 4f678044d4624ae8af21a53bac47b29d, State: Running, Role: LEADER
I20260926 14:50:22.878522 10353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:22.878597 10358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f678044d4624ae8af21a53bac47b29d" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40297 } }
I20260926 14:50:22.878791 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:22.879575 10360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f678044d4624ae8af21a53bac47b29d. Latest consensus state: current_term: 1 leader_uuid: "4f678044d4624ae8af21a53bac47b29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f678044d4624ae8af21a53bac47b29d" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40297 } } }
I20260926 14:50:22.879662 10360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:22.879978 10359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f678044d4624ae8af21a53bac47b29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f678044d4624ae8af21a53bac47b29d" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40297 } } }
I20260926 14:50:22.880033 10372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:22.880124 10359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:22.880479 10372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:22.881872 10372 catalog_manager.cc:1347] Generated new cluster ID: 197f6d44f90a44eb8860d126b49ec357
I20260926 14:50:22.881925 10372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:22.897266 10372 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:22.897938 10372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:22.905833 10372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f678044d4624ae8af21a53bac47b29d: Generated new TSK 0
I20260926 14:50:22.906088 10372 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:50:22.938661 10383 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 14:50:22.938778 10383 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:50:22.938943 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:22.916535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.5.15.190:40297)
0926 14:50:22.916597 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:22.916600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:22.916735 (+ 135us) server_negotiation.cc:407] Connection header received
0926 14:50:22.916829 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:22.916830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:22.916839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:22.916889 (+ 50us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:22.917026 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.917495 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.918624 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.918862 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.918912 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.919117 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:22.938658 (+ 19541us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:50:22.938865 (+ 207us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:50:22.939019 10356 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36068: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 14:50:22.939232 10382 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:22.916484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40297 (local address 127.0.0.1:36068)
0926 14:50:22.916658 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:22.916668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:22.916744 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:22.916903 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:22.916904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:22.916906 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:22.916970 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.916972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.917570 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.917570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.918543 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:22.918546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:22.918928 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:22.918929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:22.918997 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:22.919017 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:22.938807 (+ 19790us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:50:22.938921 (+ 114us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.15.190:40297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.15.190:40297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 14:50:22.939637 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10286
2026-09-26T14:50:22Z chronyd exiting
I20260926 14:50:22.975679 5182 test_util.cc:175] -----------------------------------------------
I20260926 14:50:22.975744 5182 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434201368912-5182-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 14:50:22.976469 5182 mini_oidc.cc:152] Starting JWKS server
I20260926 14:50:22.977082 5182 webserver.cc:460] Webserver started at https://127.0.0.1:40903/ using document root <none> and password file <none>
I20260926 14:50:22.977114 5182 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:50:22.977267 5182 webserver.cc:460] Webserver started at http://127.0.0.1:46509/ using document root <none> and password file <none>
2026-09-26T14:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:22Z Disabled control of system clock
I20260926 14:50:22.984217 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40781
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:45147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:40781
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40903/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/testchainca.cert with env {}
W20260926 14:50:23.074918 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:23.075129 10389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:23.075163 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:23.076336 10389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:23.076402 10389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:23.076426 10389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:23.076459 10389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:23.076484 10389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:23.076514 10389 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:23.076540 10389 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40903/jwks.json
I20260926 14:50:23.078058 10389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:45147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:40781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40903/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:23.078353 10389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:23.078572 10389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:23.083165 10395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:23.083180 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:23.083292 10389 server_base.cc:1034] running on GCE node
W20260926 14:50:23.083180 10394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:23.083587 10389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:23.083847 10389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:23.085086 10389 hybrid_clock.cc:648] HybridClock initialized: now 1790434223085067 us; error 35 us; skew 500 ppm
I20260926 14:50:23.086643 10389 webserver.cc:460] Webserver started at http://127.5.15.190:45413/ using document root <none> and password file <none>
I20260926 14:50:23.086930 10389 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:23.086990 10389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:23.087116 10389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:23.088304 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "b11277d05fb648ff950adce83593f18a"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "2f019c3a9d1e139896e568b46c4e4134"
server_key_iv: "ed8969ce1938b78dccf99d3c069c5632"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.088668 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "b11277d05fb648ff950adce83593f18a"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "2f019c3a9d1e139896e568b46c4e4134"
server_key_iv: "ed8969ce1938b78dccf99d3c069c5632"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.089856 10389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:23.090492 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:23.090638 10389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:23.090723 10389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b11277d05fb648ff950adce83593f18a"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "2f019c3a9d1e139896e568b46c4e4134"
server_key_iv: "ed8969ce1938b78dccf99d3c069c5632"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.090837 10389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:23.130913 10389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:23.131184 10389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:23.131284 10389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:23.134946 10389 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:40781
I20260926 14:50:23.135000 10455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:40781 every 8 connection(s)
I20260926 14:50:23.135352 10389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:23.136258 10456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:23.138441 10456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a: Bootstrap starting.
I20260926 14:50:23.138778 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10389
I20260926 14:50:23.138893 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:23.139072 5182 external_mini_cluster.cc:1428] Setting key 052bb610b73439b2bccf429e46646b1e
I20260926 14:50:23.139111 10456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:23.139428 10456 log.cc:824] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:23.140172 10456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a: No bootstrap required, opened a new log
I20260926 14:50:23.142541 10456 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11277d05fb648ff950adce83593f18a" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40781 } }
I20260926 14:50:23.142670 10456 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:23.142692 10456 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11277d05fb648ff950adce83593f18a, State: Initialized, Role: FOLLOWER
I20260926 14:50:23.142848 10456 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11277d05fb648ff950adce83593f18a" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40781 } }
I20260926 14:50:23.142925 10456 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:23.142947 10456 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:23.143019 10456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:23.143685 10456 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11277d05fb648ff950adce83593f18a" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40781 } }
I20260926 14:50:23.143796 10456 leader_election.cc:302] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11277d05fb648ff950adce83593f18a; no voters:
I20260926 14:50:23.143980 10456 leader_election.cc:288] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:23.144063 10461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:23.144224 10456 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:23.144234 10461 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [term 1 LEADER]: Becoming Leader. State: Replica: b11277d05fb648ff950adce83593f18a, State: Running, Role: LEADER
I20260926 14:50:23.144387 10461 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11277d05fb648ff950adce83593f18a" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40781 } }
I20260926 14:50:23.144791 10463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11277d05fb648ff950adce83593f18a. Latest consensus state: current_term: 1 leader_uuid: "b11277d05fb648ff950adce83593f18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11277d05fb648ff950adce83593f18a" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40781 } } }
I20260926 14:50:23.144892 10463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:23.145109 10462 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11277d05fb648ff950adce83593f18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11277d05fb648ff950adce83593f18a" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40781 } } }
I20260926 14:50:23.145206 10462 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:23.145129 10470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:23.145706 10470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:23.147140 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.139580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.5.15.190:40781)
0926 14:50:23.139767 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.139776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.139809 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:50:23.140418 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.140426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.140499 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.140650 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:23.140847 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.142289 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.143254 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.143894 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.144026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.146487 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:23.146490 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:23.146490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:23.146664 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:23.146697 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:23.146707 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:23.146805 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:23.146882 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 14:50:23.146961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260926 14:50:23.147672 10470 catalog_manager.cc:1347] Generated new cluster ID: 7bf74e7ea9a642e98a5c936aff058ea8
I20260926 14:50:23.147724 10470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:23.148120 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:23.156600 10470 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:23.157238 10470 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:23.159924 10470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11277d05fb648ff950adce83593f18a: Generated new TSK 0
I20260926 14:50:23.160079 10470 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:50:23.180485 10486 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 14:50:23.180554 10486 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:50:23.180838 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.166678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.5.15.190:40781)
0926 14:50:23.166775 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.166777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.166855 (+ 78us) server_negotiation.cc:407] Connection header received
0926 14:50:23.166936 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.166937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.166946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.167015 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:23.167158 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.167602 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.168678 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.168856 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.168901 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.169081 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:23.180672 (+ 11591us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:50:23.180793 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 14:50:23.180902 10460 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 14:50:23.181010 10485 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:23.166609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:40781 (local address 127.0.0.1:51388)
0926 14:50:23.166788 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:23.166800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:23.166871 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:23.167036 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:23.167037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:23.167039 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:23.167097 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:23.167099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.167692 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.167693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:23.168601 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:23.168603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.168927 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.168928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:23.168986 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.168992 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:23.180785 (+ 11793us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:50:23.180909 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.15.190:40781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40903/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260926 14:50:23.181308 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10389
2026-09-26T14:50:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 14:50:23.218114 5182 mini_oidc.cc:152] Starting JWKS server
I20260926 14:50:23.218825 5182 webserver.cc:460] Webserver started at https://127.0.0.1:35279/ using document root <none> and password file <none>
I20260926 14:50:23.218870 5182 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:50:23.218987 5182 webserver.cc:460] Webserver started at http://127.0.0.1:45475/ using document root <none> and password file <none>
2026-09-26T14:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:23Z Disabled control of system clock
I20260926 14:50:23.226112 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:34761
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:32863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:34761
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35279/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:50:23.301028 10492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:23.301227 10492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:23.301249 10492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:23.302381 10492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:23.302420 10492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:23.302433 10492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:23.302485 10492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:23.302512 10492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:23.302531 10492 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:23.302557 10492 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35279/jwks.json
I20260926 14:50:23.303882 10492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:32863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:34761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:34761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35279/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:23.304051 10492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:23.304334 10492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:23.309110 10498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:23.309095 10497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:23.309152 10500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:23.309290 10492 server_base.cc:1034] running on GCE node
I20260926 14:50:23.309597 10492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:23.309900 10492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:23.311054 10492 hybrid_clock.cc:648] HybridClock initialized: now 1790434223311048 us; error 35 us; skew 500 ppm
I20260926 14:50:23.312248 10492 webserver.cc:460] Webserver started at http://127.5.15.190:39965/ using document root <none> and password file <none>
I20260926 14:50:23.312493 10492 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:23.312558 10492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:23.312687 10492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:23.313709 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "e741c7625aa04967a607bb4215c47f95"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "116072345f46abb0bb34b90ca798ec06"
server_key_iv: "4faffde37d3e91725b0f19be4776931a"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.314059 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "e741c7625aa04967a607bb4215c47f95"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "116072345f46abb0bb34b90ca798ec06"
server_key_iv: "4faffde37d3e91725b0f19be4776931a"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.315400 10492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:23.316067 10506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:23.316237 10492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:23.316331 10492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "e741c7625aa04967a607bb4215c47f95"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "116072345f46abb0bb34b90ca798ec06"
server_key_iv: "4faffde37d3e91725b0f19be4776931a"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.316416 10492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:23.334944 10492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:23.335181 10492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:23.335347 10492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:23.338841 10492 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:34761
I20260926 14:50:23.338863 10558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:34761 every 8 connection(s)
I20260926 14:50:23.339179 10492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:23.339771 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10492
I20260926 14:50:23.339800 10559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:23.339874 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:23.340025 5182 external_mini_cluster.cc:1428] Setting key 3b4a581e756c819a911e93268db2c62c
I20260926 14:50:23.343626 10559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95: Bootstrap starting.
I20260926 14:50:23.344254 10559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:23.344548 10559 log.cc:824] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:23.345289 10559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95: No bootstrap required, opened a new log
I20260926 14:50:23.346134 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.340485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.5.15.190:34761)
0926 14:50:23.340718 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.340730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.340766 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:50:23.341408 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.341416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.341496 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.341662 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:23.341837 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.343193 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.343990 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.344624 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.344748 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.345473 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:23.345477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:23.345477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:23.345670 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:23.345710 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:23.345723 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:23.345792 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:23.345853 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 14:50:23.345927 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260926 14:50:23.347131 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:23.347496 10559 raft_consensus.cc:348] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e741c7625aa04967a607bb4215c47f95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 34761 } }
I20260926 14:50:23.347659 10559 raft_consensus.cc:374] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:23.347695 10559 raft_consensus.cc:729] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e741c7625aa04967a607bb4215c47f95, State: Initialized, Role: FOLLOWER
I20260926 14:50:23.347833 10559 consensus_queue.cc:260] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e741c7625aa04967a607bb4215c47f95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 34761 } }
I20260926 14:50:23.347931 10559 raft_consensus.cc:388] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:23.347967 10559 raft_consensus.cc:482] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:23.348014 10559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:23.348563 10559 raft_consensus.cc:504] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e741c7625aa04967a607bb4215c47f95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 34761 } }
I20260926 14:50:23.348677 10559 leader_election.cc:302] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e741c7625aa04967a607bb4215c47f95; no voters:
I20260926 14:50:23.348850 10559 leader_election.cc:288] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:23.348887 10564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:23.349041 10564 raft_consensus.cc:686] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [term 1 LEADER]: Becoming Leader. State: Replica: e741c7625aa04967a607bb4215c47f95, State: Running, Role: LEADER
I20260926 14:50:23.349107 10559 sys_catalog.cc:564] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:23.349119 10564 consensus_queue.cc:237] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e741c7625aa04967a607bb4215c47f95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 34761 } }
I20260926 14:50:23.349501 10566 sys_catalog.cc:455] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e741c7625aa04967a607bb4215c47f95. Latest consensus state: current_term: 1 leader_uuid: "e741c7625aa04967a607bb4215c47f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e741c7625aa04967a607bb4215c47f95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 34761 } } }
I20260926 14:50:23.349589 10566 sys_catalog.cc:458] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:23.349493 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e741c7625aa04967a607bb4215c47f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e741c7625aa04967a607bb4215c47f95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 34761 } } }
I20260926 14:50:23.349707 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:23.349817 10571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:23.350268 10571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:23.351745 10571 catalog_manager.cc:1347] Generated new cluster ID: 8272a862c7254465bd28618c814fbf93
I20260926 14:50:23.351810 10571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:23.371569 10571 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:23.372249 10571 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:23.380710 10571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e741c7625aa04967a607bb4215c47f95: Generated new TSK 0
I20260926 14:50:23.380911 10571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:23.396296 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.392798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.5.15.190:34761)
0926 14:50:23.392866 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.392869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.392999 (+ 130us) server_negotiation.cc:407] Connection header received
0926 14:50:23.393059 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.393060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.393071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.393130 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:23.393263 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.393720 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.394898 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.395071 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.395122 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.395299 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:23.396069 (+ 770us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:50:23.396213 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 14:50:23.396370 10588 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:23.392738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:34761 (local address 127.0.0.1:45920)
0926 14:50:23.392921 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:23.392932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:23.392999 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:23.393144 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:23.393145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:23.393147 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:23.393201 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:23.393203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.393824 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.393825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:23.394816 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:23.394819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.395144 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.395145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:23.395205 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.395210 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:23.396187 (+ 977us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:50:23.396270 (+ 83us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.15.190:34761: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":131,"threads_started":1}
W20260926 14:50:23.396371 10562 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 14:50:23.396632 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10492
E20260926 14:50:23.407363 10589 webserver.cc:533] Webserver: sslize failed:
E20260926 14:50:23.407428 10589 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T14:50:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T14:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:23Z Disabled control of system clock
I20260926 14:50:23.440806 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44387
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:44387
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/testchainca.cert with env {}
W20260926 14:50:23.518356 10593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:23.518579 10593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:23.518615 10593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:23.519778 10593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:23.519840 10593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:23.519860 10593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:23.519879 10593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:23.519910 10593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:50:23.519933 10593 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:50:23.519958 10593 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 14:50:23.521405 10593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:44387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:23.521658 10593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:23.521864 10593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:23.526588 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:23.526597 10593 server_base.cc:1034] running on GCE node
W20260926 14:50:23.526623 10599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:23.526772 10601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:23.527020 10593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:23.527339 10593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:23.528515 10593 hybrid_clock.cc:648] HybridClock initialized: now 1790434223528499 us; error 28 us; skew 500 ppm
I20260926 14:50:23.529800 10593 webserver.cc:460] Webserver started at http://127.5.15.190:44913/ using document root <none> and password file <none>
I20260926 14:50:23.530053 10593 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:23.530119 10593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:23.530259 10593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:23.531278 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "495a8b96441f4c0c96ad3efebf48e905"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "47fe4e55cb02f446161cbc8205a4d17a"
server_key_iv: "39f556394c5943bfcb3e3ea1b0d51974"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.531620 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "495a8b96441f4c0c96ad3efebf48e905"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "47fe4e55cb02f446161cbc8205a4d17a"
server_key_iv: "39f556394c5943bfcb3e3ea1b0d51974"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.532873 10593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:23.533589 10607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:23.533751 10593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:23.533854 10593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "495a8b96441f4c0c96ad3efebf48e905"
format_stamp: "Formatted at 2026-09-26 14:50:23 on dist-test-slave-s4p1"
server_key: "47fe4e55cb02f446161cbc8205a4d17a"
server_key_iv: "39f556394c5943bfcb3e3ea1b0d51974"
server_key_version: "encryptionkey@0"
I20260926 14:50:23.533933 10593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:23.552215 10593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:23.552515 10593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:23.552663 10593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:23.556556 10593 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:44387
I20260926 14:50:23.556591 10659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:44387 every 8 connection(s)
I20260926 14:50:23.556910 10593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:23.557782 10660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:23.559983 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905: Bootstrap starting.
I20260926 14:50:23.560542 10660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:23.560828 10660 log.cc:824] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:23.561472 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905: No bootstrap required, opened a new log
I20260926 14:50:23.563040 10660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495a8b96441f4c0c96ad3efebf48e905" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44387 } }
I20260926 14:50:23.563165 10660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:23.563187 10660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495a8b96441f4c0c96ad3efebf48e905, State: Initialized, Role: FOLLOWER
I20260926 14:50:23.563362 10660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495a8b96441f4c0c96ad3efebf48e905" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44387 } }
I20260926 14:50:23.563422 10660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:23.563442 10660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:23.563508 10660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:23.564045 10660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495a8b96441f4c0c96ad3efebf48e905" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44387 } }
I20260926 14:50:23.564149 10660 leader_election.cc:302] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 495a8b96441f4c0c96ad3efebf48e905; no voters:
I20260926 14:50:23.564317 10660 leader_election.cc:288] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:23.564355 10663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:23.564507 10663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [term 1 LEADER]: Becoming Leader. State: Replica: 495a8b96441f4c0c96ad3efebf48e905, State: Running, Role: LEADER
I20260926 14:50:23.564565 10660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:23.564599 10663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495a8b96441f4c0c96ad3efebf48e905" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44387 } }
I20260926 14:50:23.565021 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495a8b96441f4c0c96ad3efebf48e905. Latest consensus state: current_term: 1 leader_uuid: "495a8b96441f4c0c96ad3efebf48e905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495a8b96441f4c0c96ad3efebf48e905" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44387 } } }
I20260926 14:50:23.565047 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10593
I20260926 14:50:23.565099 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:23.565168 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:23.565382 5182 external_mini_cluster.cc:1428] Setting key 6dd4647fe128de6c3c3696a82f8efb50
I20260926 14:50:23.565378 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "495a8b96441f4c0c96ad3efebf48e905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "495a8b96441f4c0c96ad3efebf48e905" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44387 } } }
I20260926 14:50:23.565435 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:23.566398 10680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:23.566452 10680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:23.566494 10670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:23.566655 10670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:23.568113 10670 catalog_manager.cc:1347] Generated new cluster ID: 18c1de00b9b2437a9f49c6eaae369ede
I20260926 14:50:23.568166 10670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:23.571185 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.566013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.5.15.190:44387)
0926 14:50:23.566377 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.566384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.566408 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:50:23.566533 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.566541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.566623 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.566799 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:23.567108 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.568413 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.569184 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.569602 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.569746 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.570460 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:23.570464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:23.570464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:23.570634 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:23.570666 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:23.570677 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:23.570747 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:23.570950 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:50:23.571026 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":62,"threads_started":1}
I20260926 14:50:23.571974 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:50:23.588917 10670 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:23.589607 10670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:23.603142 10670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495a8b96441f4c0c96ad3efebf48e905: Generated new TSK 0
I20260926 14:50:23.603340 10670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:24.056936 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:23.612908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.5.15.190:44387)
0926 14:50:23.612968 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:23.612970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:23.613059 (+ 89us) server_negotiation.cc:407] Connection header received
0926 14:50:23.613124 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:23.613124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:23.613137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:23.613205 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:50:23.613305 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.614120 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.614922 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.615240 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.615319 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.615467 (+ 148us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:50:24.056707 (+441240us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:50:24.056875 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38396: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 14:50:24.057029 10681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38396: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 14:50:24.057040 10689 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:23.612819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44387 (local address 127.0.0.1:38396)
0926 14:50:23.612997 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:23.613006 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:23.613056 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:23.613209 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:23.613210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:23.613212 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:50:23.613263 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:23.613265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.614188 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.614189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:23.614842 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:23.614845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:23.615329 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:23.615330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:23.615376 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:23.615381 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:50:24.056850 (+441469us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:50:24.056966 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.15.190:44387: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260926 14:50:24.057722 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10593
2026-09-26T14:50:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (641 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:24 dist-test-slave-s4p1 krb5kdc[10701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:24 dist-test-slave-s4p1 krb5kdc[10701](info): set up 2 sockets
Sep 26 14:50:24 dist-test-slave-s4p1 krb5kdc[10701](info): commencing operation
krb5kdc: starting...
W20260926 14:50:25.254820 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:25.366178 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:43495
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:25.444828 10714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:25.445029 10714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:25.445050 10714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:25.446154 10714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:25.446187 10714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:25.446201 10714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:25.446214 10714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:25.446226 10714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:25.447607 10714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:25.447790 10714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:25.447948 10714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:25.452915 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.452936 10719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:25.452970 10714 server_base.cc:1034] running on GCE node
W20260926 14:50:25.452898 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:25.453316 10714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:25.453605 10714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:25.454793 10714 hybrid_clock.cc:648] HybridClock initialized: now 1790434225454771 us; error 37 us; skew 500 ppm
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:25.457268 10714 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:25.457625 10714 webserver.cc:460] Webserver started at http://127.5.15.190:45949/ using document root <none> and password file <none>
I20260926 14:50:25.457788 10714 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:25.457834 10714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:25.457917 10714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:25.458956 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "042cfa878c974f16bc8ee621ae792903"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "5bb67dbecd570a5a7c85ea6409793e2e"
server_key_iv: "16c642826ae97436ff9caaecccccc51c"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.459254 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "042cfa878c974f16bc8ee621ae792903"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "5bb67dbecd570a5a7c85ea6409793e2e"
server_key_iv: "16c642826ae97436ff9caaecccccc51c"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.460507 10714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:25.461130 10729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.461294 10714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:25.461393 10714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "042cfa878c974f16bc8ee621ae792903"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "5bb67dbecd570a5a7c85ea6409793e2e"
server_key_iv: "16c642826ae97436ff9caaecccccc51c"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.461468 10714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:25.471935 10714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:25.474187 10714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:25.474342 10714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:25.478091 10714 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:43495
I20260926 14:50:25.478129 10781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:43495 every 8 connection(s)
I20260926 14:50:25.478494 10714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:25.479363 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:25.480370 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10714
I20260926 14:50:25.480484 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:25.480665 5182 external_mini_cluster.cc:1428] Setting key 719c5794e77d207056afc04e23531404
I20260926 14:50:25.482105 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903: Bootstrap starting.
I20260926 14:50:25.482689 10782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:25.483053 10782 log.cc:824] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:25.483778 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903: No bootstrap required, opened a new log
I20260926 14:50:25.485868 10782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042cfa878c974f16bc8ee621ae792903" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 43495 } }
I20260926 14:50:25.486037 10782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:25.486078 10782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042cfa878c974f16bc8ee621ae792903, State: Initialized, Role: FOLLOWER
I20260926 14:50:25.486258 10782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042cfa878c974f16bc8ee621ae792903" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 43495 } }
I20260926 14:50:25.486351 10782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:25.486395 10782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:25.486445 10782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:25.487036 10782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042cfa878c974f16bc8ee621ae792903" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 43495 } }
I20260926 14:50:25.487154 10782 leader_election.cc:302] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 042cfa878c974f16bc8ee621ae792903; no voters:
I20260926 14:50:25.487367 10782 leader_election.cc:288] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:25.487390 10787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:25.487531 10787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [term 1 LEADER]: Becoming Leader. State: Replica: 042cfa878c974f16bc8ee621ae792903, State: Running, Role: LEADER
I20260926 14:50:25.487655 10782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:25.487663 10787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042cfa878c974f16bc8ee621ae792903" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 43495 } }
I20260926 14:50:25.488094 10788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "042cfa878c974f16bc8ee621ae792903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042cfa878c974f16bc8ee621ae792903" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 43495 } } }
I20260926 14:50:25.488209 10788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:25.489404 10789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042cfa878c974f16bc8ee621ae792903. Latest consensus state: current_term: 1 leader_uuid: "042cfa878c974f16bc8ee621ae792903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042cfa878c974f16bc8ee621ae792903" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 43495 } } }
W20260926 14:50:25.489538 10803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 14:50:25.489553 10789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:25.489639 10803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:25.490105 10795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:25.490291 10795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:25.491715 10795 catalog_manager.cc:1347] Generated new cluster ID: 9a18145f5b734e1298bc7b4555258e06
I20260926 14:50:25.491771 10795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:25.492286 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:25.481171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.5.15.190:43495)
0926 14:50:25.481356 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:25.481366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:25.481400 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:25.481914 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:25.481922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:25.481994 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:25.482147 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:25.482803 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.484135 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.484831 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.485471 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.485603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:25.488331 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:25.488336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:25.488337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:25.488347 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:25.490585 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.490999 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.491002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.491002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.491101 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.491432 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.491433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.491434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.491700 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:25.491848 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:25.491922 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 14:50:25.492054 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260926 14:50:25.501688 10795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:25.502334 10795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:25.511664 10795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 042cfa878c974f16bc8ee621ae792903: Generated new TSK 0
I20260926 14:50:25.511825 10795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:25.543407 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:43495
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:25.621439 10808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:25.621618 10808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:25.621639 10808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:25.622648 10808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:25.622676 10808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:25.622699 10808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:25.624035 10808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:43495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:25.624200 10808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:25.624368 10808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:25.629004 10813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.629048 10816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.629052 10814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:25.629158 10808 server_base.cc:1034] running on GCE node
I20260926 14:50:25.629492 10808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:25.629751 10808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:25.630872 10808 hybrid_clock.cc:648] HybridClock initialized: now 1790434225630848 us; error 38 us; skew 500 ppm
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:25.633345 10808 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:25.633777 10808 webserver.cc:460] Webserver started at http://127.5.15.129:37113/ using document root <none> and password file <none>
I20260926 14:50:25.633991 10808 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:25.634055 10808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:25.634203 10808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:25.635267 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "5049e57d801f47cb9ef3323b29eccce6"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "4d652649fabfbe9d870915d354fff8c3"
server_key_iv: "f08898bf010a623bc4e3cb056098da67"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.635648 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "5049e57d801f47cb9ef3323b29eccce6"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "4d652649fabfbe9d870915d354fff8c3"
server_key_iv: "f08898bf010a623bc4e3cb056098da67"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.636916 10808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:25.637588 10823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.637748 10808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:25.637842 10808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "5049e57d801f47cb9ef3323b29eccce6"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "4d652649fabfbe9d870915d354fff8c3"
server_key_iv: "f08898bf010a623bc4e3cb056098da67"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.637923 10808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:25.648308 10808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:25.650727 10808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:25.650974 10808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:25.651203 10808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:25.651506 10808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:25.651561 10808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.651611 10808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:25.651638 10808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.657112 10808 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:36625
I20260926 14:50:25.657155 10936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:36625 every 8 connection(s)
I20260926 14:50:25.657469 10808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:25.666577 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:25.658130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:46517 (local address 127.5.15.190:43495)
0926 14:50:25.658186 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:25.658189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:25.658390 (+ 201us) server_negotiation.cc:407] Connection header received
0926 14:50:25.659381 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:25.659383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:25.659394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:25.659450 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:25.660009 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.660502 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.661456 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.661677 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.661753 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:25.663753 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:25.663755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:25.663755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:25.663763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:25.665212 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.665623 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.665625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.665625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.665680 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.665938 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.665938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.665939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.666057 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:25.666132 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:25.666403 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 14:50:25.666488 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:25.666738 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:25.658082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:43495 (local address 127.5.15.129:46517)
0926 14:50:25.658273 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:25.658310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:25.659256 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:25.659528 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:25.659531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:25.659742 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:25.659908 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:25.659912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.660598 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.660599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:25.661365 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:25.661367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.661758 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.661760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:25.661874 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:25.662370 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:25.662378 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:25.663657 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:25.665326 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:25.665333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:25.665335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:25.665545 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:25.665754 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:25.665755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:25.665756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:25.665863 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:25.666125 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:25.666131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:25.666264 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 14:50:25.666371 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:50:25.666516 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260926 14:50:25.667241 10937 heartbeater.cc:347] Connected to a master server at 127.5.15.190:43495
I20260926 14:50:25.667344 10937 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:25.667516 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10808
I20260926 14:50:25.667532 10937 heartbeater.cc:511] Master 127.5.15.190:43495 requested a full tablet report, sending...
I20260926 14:50:25.667616 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:25.667809 5182 external_mini_cluster.cc:1428] Setting key 674f0c63d09594b7ad233ff97ed5d2e9
I20260926 14:50:25.668063 10746 ts_manager.cc:194] Registered new tserver with Master: 5049e57d801f47cb9ef3323b29eccce6 (127.5.15.129:36625)
I20260926 14:50:25.668785 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:46517
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:25.691643 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:43495
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:25.769316 10942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:25.769516 10942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:25.769538 10942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:25.770725 10942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:25.770805 10942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:25.770864 10942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:25.772302 10942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:43495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:25.772524 10942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:25.772742 10942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:25.777626 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.777652 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.777626 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:25.777842 10942 server_base.cc:1034] running on GCE node
I20260926 14:50:25.778056 10942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:25.778258 10942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:25.779417 10942 hybrid_clock.cc:648] HybridClock initialized: now 1790434225779407 us; error 26 us; skew 500 ppm
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:25.781869 10942 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:25.782328 10942 webserver.cc:460] Webserver started at http://127.5.15.130:33245/ using document root <none> and password file <none>
I20260926 14:50:25.782563 10942 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:25.782631 10942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:25.782828 10942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:25.783833 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "8004c3606c79472ba1f846398089eaf3"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "6d387def18aaf70b38e28af4cdaa4cf2"
server_key_iv: "1a3e557442002f1e2890268a9e8c033b"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.784198 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "8004c3606c79472ba1f846398089eaf3"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "6d387def18aaf70b38e28af4cdaa4cf2"
server_key_iv: "1a3e557442002f1e2890268a9e8c033b"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.785476 10942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:25.786171 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.786332 10942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:25.786427 10942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "8004c3606c79472ba1f846398089eaf3"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "6d387def18aaf70b38e28af4cdaa4cf2"
server_key_iv: "1a3e557442002f1e2890268a9e8c033b"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.786505 10942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:25.799642 10942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:25.801937 10942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:25.802114 10942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:25.802356 10942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:25.802660 10942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:25.802714 10942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.802778 10942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:25.802809 10942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.808486 10942 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:37249
I20260926 14:50:25.808533 11070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:37249 every 8 connection(s)
I20260926 14:50:25.808835 10942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:25.816100 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 10942
I20260926 14:50:25.816215 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:25.816429 5182 external_mini_cluster.cc:1428] Setting key 471257c53280dd2112c8a0dee78066d8
I20260926 14:50:25.818125 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:25.809574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:57275 (local address 127.5.15.190:43495)
0926 14:50:25.809642 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:25.809645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:25.809856 (+ 211us) server_negotiation.cc:407] Connection header received
0926 14:50:25.810728 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:25.810729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:25.810738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:25.810846 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:25.811468 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.811951 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.812917 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.813147 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.813201 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:25.815137 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:25.815138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:25.815139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:25.815146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:25.816662 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.817163 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.817164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.817164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.817227 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:25.817516 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:25.817516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:25.817517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:25.817618 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:25.817668 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:25.817986 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 14:50:25.818048 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:25.818441 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:25.809551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:43495 (local address 127.5.15.130:57275)
0926 14:50:25.809735 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:25.809769 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:25.810640 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:25.810911 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:25.810914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:25.811151 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:25.811376 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:25.811398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.812024 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.812024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:25.812801 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:25.812803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:25.813238 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:25.813239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:25.813340 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:25.813758 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:25.813766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:25.815053 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:25.816778 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:25.816784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:25.816786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:25.817069 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:25.817303 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:25.817304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:25.817305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:25.817450 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:25.817681 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:25.817687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:25.817855 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 14:50:25.817965 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:50:25.818155 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260926 14:50:25.818931 11071 heartbeater.cc:347] Connected to a master server at 127.5.15.190:43495
I20260926 14:50:25.819027 11071 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:25.819209 11071 heartbeater.cc:511] Master 127.5.15.190:43495 requested a full tablet report, sending...
I20260926 14:50:25.819582 10746 ts_manager.cc:194] Registered new tserver with Master: 8004c3606c79472ba1f846398089eaf3 (127.5.15.130:37249)
I20260926 14:50:25.820129 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:57275
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:25.839747 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:43495
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:25.918404 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:25.918591 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:25.918612 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:25.919788 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:25.919831 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:25.919857 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:25.921329 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:43495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:25.921550 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:25.921746 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:25.927078 11084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.927090 11082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:25.927088 11081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:25.927268 11076 server_base.cc:1034] running on GCE node
I20260926 14:50:25.927474 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:25.927670 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:25.928835 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790434225928819 us; error 34 us; skew 500 ppm
Sep 26 14:50:25 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:25.931221 11076 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:25.931643 11076 webserver.cc:460] Webserver started at http://127.5.15.131:43283/ using document root <none> and password file <none>
I20260926 14:50:25.931864 11076 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:25.931907 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:25.932049 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:25.933007 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "e1e832ac27b14a4f9a991b48edb7adc8"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "f1c313f5d40cb94208c1eb96b2da8cbf"
server_key_iv: "90982447a47f9d094b227346a6c1a994"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.933347 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "e1e832ac27b14a4f9a991b48edb7adc8"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "f1c313f5d40cb94208c1eb96b2da8cbf"
server_key_iv: "90982447a47f9d094b227346a6c1a994"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.934548 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:25.935232 11091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.935389 11076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:25.935479 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "e1e832ac27b14a4f9a991b48edb7adc8"
format_stamp: "Formatted at 2026-09-26 14:50:25 on dist-test-slave-s4p1"
server_key: "f1c313f5d40cb94208c1eb96b2da8cbf"
server_key_iv: "90982447a47f9d094b227346a6c1a994"
server_key_version: "encryptionkey@0"
I20260926 14:50:25.935555 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:26.000276 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:26.002202 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:26.002362 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:26.002616 11076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:26.002961 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:26.003016 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.003058 11076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:26.003082 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:26.008584 11076 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:39459
I20260926 14:50:26.008631 11204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:39459 every 8 connection(s)
I20260926 14:50:26.008988 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:26.015271 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11076
I20260926 14:50:26.015357 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:26.015544 5182 external_mini_cluster.cc:1428] Setting key dbe939dffe26936822ebc1bc98f0a695
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434225, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:26.018497 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.009704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:54747 (local address 127.5.15.190:43495)
0926 14:50:26.009789 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.009791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.010010 (+ 219us) server_negotiation.cc:407] Connection header received
0926 14:50:26.010985 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.010986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.010996 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.011050 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.011823 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.012420 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.013600 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.013799 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.013873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.016119 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.016120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.016121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.016127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.017451 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.017780 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.017781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.017781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.017828 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.017994 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.017994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.017994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.018086 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.018147 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.018373 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.018435 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:50:26.018651 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.009667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:43495 (local address 127.5.15.131:54747)
0926 14:50:26.009856 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:26.009909 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:26.010888 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:26.011110 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:26.011113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:26.011502 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:26.011688 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.011693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.012499 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.012499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.013332 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:26.013334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.013890 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.013890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:26.013997 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.014493 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:26.014501 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:26.016026 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:26.017513 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.017519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.017521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.017721 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.017868 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:26.017868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:26.017869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:26.017947 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:26.018154 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:26.018158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:26.018263 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 14:50:26.018347 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:50:26.018458 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260926 14:50:26.019124 11205 heartbeater.cc:347] Connected to a master server at 127.5.15.190:43495
I20260926 14:50:26.019207 11205 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:26.019356 11205 heartbeater.cc:511] Master 127.5.15.190:43495 requested a full tablet report, sending...
I20260926 14:50:26.019680 10746 ts_manager.cc:194] Registered new tserver with Master: e1e832ac27b14a4f9a991b48edb7adc8 (127.5.15.131:39459)
I20260926 14:50:26.020051 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:54747
I20260926 14:50:26.027298 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/test-admin.keytab.
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.043072 5182 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 14:50:26.044654 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:26.043883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.5.15.129:36625)
0926 14:50:26.044076 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.044081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.044090 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:26.044116 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.044118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.044132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.044201 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.044546 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52106: BlockingRecv error: recv got EOF from 127.0.0.1:52106 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
W20260926 14:50:26.044761 5182 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.045749 5182 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.046643 5182 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:26.053929 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.047158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.5.15.129:36625)
0926 14:50:26.047205 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.047207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.047222 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:26.047278 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.047279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.047287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.047382 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.047726 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.048488 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.049150 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.049442 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.049491 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.051474 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.051481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.051482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.051488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.053080 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.053368 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.053370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.053371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.053419 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.053550 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.053550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.053550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.053653 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.053737 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.053807 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.053863 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 14:50:26.055716 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:26.055154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.5.15.129:36625)
0926 14:50:26.055204 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.055206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.055250 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:50:26.055328 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.055329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.055339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.055381 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.055663 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52134: BlockingRecv error: recv got EOF from 127.0.0.1:52134 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 14:50:26.055778 5182 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.056664 5182 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.057490 5182 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:26.064531 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.057971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.5.15.129:36625)
0926 14:50:26.058025 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.058027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.058094 (+ 67us) server_negotiation.cc:407] Connection header received
0926 14:50:26.058181 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.058181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.058190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.058243 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.058616 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.059157 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.059863 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.060077 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.060160 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.062080 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.062081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.062082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.062089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.063468 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.063803 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.063805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.063805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.063859 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.064047 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.064047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.064048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.064140 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.064255 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.064374 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.064445 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:50:26.065838 11212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:26.065411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.5.15.129:36625)
0926 14:50:26.065442 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.065444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.065491 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:50:26.065534 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.065534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.065541 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.065591 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.065778 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52144: BlockingRecv error: recv got EOF from 127.0.0.1:52144 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 14:50:26.065953 5182 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434201368912-5182-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.066900 5182 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:26.067695 5182 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[10701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:50:26.074468 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:26.068161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.5.15.129:36625)
0926 14:50:26.068211 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:26.068213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:26.068254 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:50:26.068337 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:26.068337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:26.068345 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:26.068410 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:26.068756 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.069220 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.069897 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:26.070082 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:26.070168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:26.072111 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:26.072112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:26.072112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:26.072119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:26.073425 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.073764 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.073765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.073765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.073822 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:26.074022 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:26.074022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:26.074023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:26.074114 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:26.074203 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:26.074312 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 14:50:26.074380 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:50:26.075109 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10808
I20260926 14:50:26.079699 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10942
I20260926 14:50:26.083841 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11076
I20260926 14:50:26.087785 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 10714
2026-09-26T14:50:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1858 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[11225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[11225](info): set up 2 sockets
Sep 26 14:50:26 dist-test-slave-s4p1 krb5kdc[11225](info): commencing operation
krb5kdc: starting...
W20260926 14:50:27.107138 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab.
Entry for principal oryx/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab.
I20260926 14:50:27.217717 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38521
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab
--principal=oryx/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:27.295961 11238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.296155 11238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.296177 11238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.297262 11238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:27.297294 11238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.297308 11238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.297322 11238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:27.297333 11238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:27.298731 11238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab
--principal=oryx/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:27.298974 11238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.299127 11238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.303638 11244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.303629 11246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.303628 11243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.303643 11238 server_base.cc:1034] running on GCE node
I20260926 14:50:27.304059 11238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.304282 11238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.305442 11238 hybrid_clock.cc:648] HybridClock initialized: now 1790434227305418 us; error 36 us; skew 500 ppm
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:27.307992 11238 init.cc:377] Logged in from keytab as oryx/127.5.15.190@KRBTEST.COM (short username oryx)
I20260926 14:50:27.308431 11238 webserver.cc:460] Webserver started at http://127.5.15.190:45915/ using document root <none> and password file <none>
I20260926 14:50:27.308640 11238 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.308702 11238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.308846 11238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.309789 11238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "31006d7f535a4a9f8ac7b9faf17be594"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "1b3b14fd698dd068041ef2c0a9f97c04"
server_key_iv: "96389c624bebfc1b990493e6e07f92a1"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.310156 11238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "31006d7f535a4a9f8ac7b9faf17be594"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "1b3b14fd698dd068041ef2c0a9f97c04"
server_key_iv: "96389c624bebfc1b990493e6e07f92a1"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.311509 11238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:27.312175 11253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.312368 11238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.312466 11238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "31006d7f535a4a9f8ac7b9faf17be594"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "1b3b14fd698dd068041ef2c0a9f97c04"
server_key_iv: "96389c624bebfc1b990493e6e07f92a1"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.312541 11238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.321382 11238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.323192 11238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.323340 11238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.327143 11238 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:38521
I20260926 14:50:27.327172 11305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:38521 every 8 connection(s)
I20260926 14:50:27.327508 11238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:27.328377 11306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:27.330384 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Bootstrap starting.
I20260926 14:50:27.330956 11306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:27.331257 11306 log.cc:824] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:27.331705 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11238
I20260926 14:50:27.331842 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:27.332039 5182 external_mini_cluster.cc:1428] Setting key 31113ed743a7fa422e34d8ea83d3562e
I20260926 14:50:27.332052 11306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: No bootstrap required, opened a new log
I20260926 14:50:27.334157 11306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } }
I20260926 14:50:27.334328 11306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:27.334370 11306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31006d7f535a4a9f8ac7b9faf17be594, State: Initialized, Role: FOLLOWER
I20260926 14:50:27.334507 11306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } }
I20260926 14:50:27.334576 11306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:27.334604 11306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:27.334645 11306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:27.335304 11306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } }
I20260926 14:50:27.335402 11306 leader_election.cc:302] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31006d7f535a4a9f8ac7b9faf17be594; no voters:
I20260926 14:50:27.335529 11306 leader_election.cc:288] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:27.335587 11311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:27.335728 11311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 LEADER]: Becoming Leader. State: Replica: 31006d7f535a4a9f8ac7b9faf17be594, State: Running, Role: LEADER
I20260926 14:50:27.335731 11306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:27.336792 11311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } }
I20260926 14:50:27.337566 11313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } }
I20260926 14:50:27.337671 11313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:27.337693 11325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:27.337738 11325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:27.337567 11312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31006d7f535a4a9f8ac7b9faf17be594. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } }
I20260926 14:50:27.337827 11312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:27.338230 11327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:27.338392 11327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:27.339884 11327 catalog_manager.cc:1347] Generated new cluster ID: 9b92355240654b549ff0a0d9722fbcca
I20260926 14:50:27.339924 11327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:27.343907 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.332521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.5.15.190:38521)
0926 14:50:27.332718 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.332726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.332761 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:50:27.333412 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.333421 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.333477 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.333637 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.334165 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.335794 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.336690 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.337298 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.337799 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.340381 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.340384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.340385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:27.340395 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.342287 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.342627 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.342630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.342630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.342728 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.342952 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.342953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.342953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.343220 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.343374 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.343440 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.343556 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260926 14:50:27.359442 11327 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:27.359941 11327 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:27.362870 11327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Generated new TSK 0
I20260926 14:50:27.363029 11327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab.
Entry for principal oryx/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab.
I20260926 14:50:27.393590 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38521
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:27.471601 11332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.471786 11332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.471809 11332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.472863 11332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.472899 11332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.472924 11332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:27.474256 11332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:27.474429 11332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.474592 11332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.479332 11338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.479331 11337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.479328 11340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.479458 11332 server_base.cc:1034] running on GCE node
I20260926 14:50:27.479727 11332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.479930 11332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.481065 11332 hybrid_clock.cc:648] HybridClock initialized: now 1790434227481052 us; error 28 us; skew 500 ppm
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:27.483448 11332 init.cc:377] Logged in from keytab as oryx/127.5.15.129@KRBTEST.COM (short username oryx)
I20260926 14:50:27.483803 11332 webserver.cc:460] Webserver started at http://127.5.15.129:33029/ using document root <none> and password file <none>
I20260926 14:50:27.483983 11332 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.484021 11332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.484105 11332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.485036 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "787cbee12fb34ed49903870502dcd5e0"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "f68be12d27d86ea0ccdbd275f8b7c531"
server_key_iv: "5b29b8195b15a98c3ffbaa34164d2b58"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.485345 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "787cbee12fb34ed49903870502dcd5e0"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "f68be12d27d86ea0ccdbd275f8b7c531"
server_key_iv: "5b29b8195b15a98c3ffbaa34164d2b58"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.486539 11332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:27.487238 11347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.487413 11332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.487509 11332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "787cbee12fb34ed49903870502dcd5e0"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "f68be12d27d86ea0ccdbd275f8b7c531"
server_key_iv: "5b29b8195b15a98c3ffbaa34164d2b58"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.487619 11332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.523053 11332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.524823 11332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.524986 11332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.525236 11332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:27.525555 11332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:27.525609 11332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.525642 11332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:27.525687 11332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.531302 11332 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:46617
I20260926 14:50:27.531350 11460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:46617 every 8 connection(s)
I20260926 14:50:27.531689 11332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:27.538064 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11332
I20260926 14:50:27.538182 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:27.538394 5182 external_mini_cluster.cc:1428] Setting key dca1cb070df2448ae6f1f85fd29def1b
I20260926 14:50:27.541049 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.532430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:57701 (local address 127.5.15.190:38521)
0926 14:50:27.532472 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.532474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.532667 (+ 193us) server_negotiation.cc:407] Connection header received
0926 14:50:27.533129 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.533130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.533139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.533205 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.534058 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.534554 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.535488 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.535687 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.535784 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.537927 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.537929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.537929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:27.537937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.539708 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.540076 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.540077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.540077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.540135 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.540378 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.540378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.540379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.540512 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.540595 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.540924 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.540987 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:27.541329 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.532396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:57701)
0926 14:50:27.532542 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.532583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.533048 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.533264 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.533267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.533802 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.533982 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.533986 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.534630 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.534630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.535416 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.535417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.535787 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.535788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.535909 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.536327 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:27.536335 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.537829 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.539798 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.539804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.539805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.540006 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.540203 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.540203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.540203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.540318 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.540597 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.540603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.540762 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.540903 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.541070 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260926 14:50:27.541832 11461 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38521
I20260926 14:50:27.541940 11461 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:27.542104 11461 heartbeater.cc:511] Master 127.5.15.190:38521 requested a full tablet report, sending...
I20260926 14:50:27.542649 11270 ts_manager.cc:194] Registered new tserver with Master: 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617)
I20260926 14:50:27.543254 11270 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:57701
WARNING: no policy specified for oryx/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab.
Entry for principal oryx/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab.
I20260926 14:50:27.561579 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38521
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:27.639014 11466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.639247 11466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.639289 11466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.640401 11466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.640462 11466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.640506 11466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:27.641924 11466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:27.642179 11466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.642387 11466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.647017 11472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.647089 11471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.647126 11466 server_base.cc:1034] running on GCE node
W20260926 14:50:27.647027 11474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.647421 11466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.647702 11466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.648881 11466 hybrid_clock.cc:648] HybridClock initialized: now 1790434227648865 us; error 24 us; skew 500 ppm
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:27.651384 11466 init.cc:377] Logged in from keytab as oryx/127.5.15.130@KRBTEST.COM (short username oryx)
I20260926 14:50:27.651775 11466 webserver.cc:460] Webserver started at http://127.5.15.130:43565/ using document root <none> and password file <none>
I20260926 14:50:27.651980 11466 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.652022 11466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.652158 11466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.653100 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "37ee595e906f467a990314e743cecc93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "00f2c649c331a66b267fbcf6b62075ef"
server_key_iv: "cb59f1f4d8df0f0dc94f2a3ae564ef9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.653445 11466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "37ee595e906f467a990314e743cecc93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "00f2c649c331a66b267fbcf6b62075ef"
server_key_iv: "cb59f1f4d8df0f0dc94f2a3ae564ef9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.654672 11466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:27.655378 11481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.655566 11466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:27.655656 11466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "37ee595e906f467a990314e743cecc93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "00f2c649c331a66b267fbcf6b62075ef"
server_key_iv: "cb59f1f4d8df0f0dc94f2a3ae564ef9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.655736 11466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.669960 11466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.671662 11466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.671808 11466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.672019 11466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:27.672307 11466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:27.672360 11466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.672397 11466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:27.672423 11466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.677846 11466 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:35939
I20260926 14:50:27.677917 11594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:35939 every 8 connection(s)
I20260926 14:50:27.678232 11466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:27.686075 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11466
I20260926 14:50:27.686182 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:27.686406 5182 external_mini_cluster.cc:1428] Setting key 2ad8ec63e91b8c410c5596dc9c0a5fc5
I20260926 14:50:27.686564 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.678963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:54447 (local address 127.5.15.190:38521)
0926 14:50:27.679004 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.679007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.679168 (+ 161us) server_negotiation.cc:407] Connection header received
0926 14:50:27.679611 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.679612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.679622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.679669 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.680465 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.680929 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.681791 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.681958 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.682005 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.683859 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.683860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.683861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:27.683867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.685247 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.685580 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.685581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.685581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.685626 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.685850 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.685851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.685851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.685939 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.685985 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.686371 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.686430 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:27.686730 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.678934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:54447)
0926 14:50:27.679073 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.679104 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.679529 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.679717 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.679719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.680230 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.680395 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.680398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.680999 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.681000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.681721 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.681723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.682022 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.682023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.682110 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.682508 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:27.682515 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.683780 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.685311 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.685316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.685317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.685508 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.685680 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.685680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.685680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.685784 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.685997 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.686003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.686158 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.686350 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.686507 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260926 14:50:27.687220 11595 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38521
I20260926 14:50:27.687328 11595 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:27.687502 11595 heartbeater.cc:511] Master 127.5.15.190:38521 requested a full tablet report, sending...
I20260926 14:50:27.687860 11270 ts_manager.cc:194] Registered new tserver with Master: 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:27.688218 11270 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:54447
WARNING: no policy specified for oryx/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab.
Entry for principal oryx/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab.
I20260926 14:50:27.711052 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38521
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:27.787397 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.787582 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.787603 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.788690 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.788728 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.788779 11600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:27.790126 11600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:27.790340 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.790566 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:27.795205 11600 server_base.cc:1034] running on GCE node
W20260926 14:50:27.795142 11605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.795223 11606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.795135 11608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.795611 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.795850 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.796984 11600 hybrid_clock.cc:648] HybridClock initialized: now 1790434227796973 us; error 27 us; skew 500 ppm
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:27.799464 11600 init.cc:377] Logged in from keytab as oryx/127.5.15.131@KRBTEST.COM (short username oryx)
I20260926 14:50:27.799849 11600 webserver.cc:460] Webserver started at http://127.5.15.131:34049/ using document root <none> and password file <none>
I20260926 14:50:27.800065 11600 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.800129 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.800277 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.801206 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "29017a39698c6cc7e2a6e634a6a5d034"
server_key_iv: "e9575748f43f0dcca5ec900b9f2a3a03"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.801533 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "29017a39698c6cc7e2a6e634a6a5d034"
server_key_iv: "e9575748f43f0dcca5ec900b9f2a3a03"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.802712 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:27.803444 11615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.803601 11600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:27.803694 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "29017a39698c6cc7e2a6e634a6a5d034"
server_key_iv: "e9575748f43f0dcca5ec900b9f2a3a03"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.803774 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.832479 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.834231 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.834380 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.834575 11600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:27.834935 11600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:27.834980 11600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.835019 11600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:27.835045 11600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.840947 11600 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:37389
I20260926 14:50:27.840991 11728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:37389 every 8 connection(s)
I20260926 14:50:27.841310 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:27.845351 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11600
I20260926 14:50:27.845441 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:27.845618 5182 external_mini_cluster.cc:1428] Setting key 032b501343a646edc88ccc1e8c8ffa1e
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:27.849747 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.841948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:44059 (local address 127.5.15.190:38521)
0926 14:50:27.842003 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.842005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.842201 (+ 196us) server_negotiation.cc:407] Connection header received
0926 14:50:27.842629 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.842630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.842639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.842694 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.843613 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.844194 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.845113 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.845296 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.845346 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.847581 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.847582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.847582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:27.847588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.848685 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.849035 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.849036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.849036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.849082 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.849276 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.849277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.849277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.849368 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.849415 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.849622 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.849687 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:50:27.849928 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.841946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:44059)
0926 14:50:27.842085 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.842116 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.842552 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.842793 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.842797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.843324 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.843494 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.843498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.844271 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.844271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.845042 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.845044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.845389 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.845391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.845548 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.846038 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:27.846047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.847504 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.848772 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.848777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.848779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.848973 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.849138 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.849138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.849139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.849225 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.849419 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.849423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.849520 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.849597 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.849703 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260926 14:50:27.850358 11729 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38521
I20260926 14:50:27.850452 11729 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:27.850607 11729 heartbeater.cc:511] Master 127.5.15.190:38521 requested a full tablet report, sending...
I20260926 14:50:27.850996 11270 ts_manager.cc:194] Registered new tserver with Master: 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389)
I20260926 14:50:27.851359 11270 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:44059
I20260926 14:50:27.857326 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.15.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.189.keytab.
Entry for principal oryx/127.5.15.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.189.keytab.
I20260926 14:50:27.879805 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:43051
--webserver_interface=127.5.15.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.189.keytab
--principal=oryx/127.5.15.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38521,127.5.15.189:43051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:27.957804 11734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:27.958030 11734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:27.958087 11734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:27.959379 11734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:27.959438 11734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:27.959462 11734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:27.959479 11734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:27.959498 11734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:27.960956 11734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38521,127.5.15.189:43051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.189.keytab
--principal=oryx/127.5.15.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:43051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.15.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
W20260926 14:50:27.961103 11734 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.15.190:38521,127.5.15.189:43051'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 14:50:27.961256 11734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:27.961462 11734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:27.965921 11740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:27.965957 11739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.966075 11734 server_base.cc:1034] running on GCE node
W20260926 14:50:27.965932 11742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:27.966426 11734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:27.966712 11734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:27.967890 11734 hybrid_clock.cc:648] HybridClock initialized: now 1790434227967871 us; error 38 us; skew 500 ppm
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:27.970093 11734 init.cc:377] Logged in from keytab as oryx/127.5.15.189@KRBTEST.COM (short username oryx)
I20260926 14:50:27.970487 11734 webserver.cc:460] Webserver started at http://127.5.15.189:39915/ using document root <none> and password file <none>
I20260926 14:50:27.970705 11734 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:27.970795 11734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:27.970938 11734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:27.971895 11734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/instance:
uuid: "cdf14746a0d94d4488b627b215cf1100"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "019f5812483c6856c210c92629441794"
server_key_iv: "256ee13a540162296a7802757b859883"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.972234 11734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal/instance:
uuid: "cdf14746a0d94d4488b627b215cf1100"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "019f5812483c6856c210c92629441794"
server_key_iv: "256ee13a540162296a7802757b859883"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.973428 11734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:50:27.974076 11749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:27.974237 11734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:27.974329 11734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
uuid: "cdf14746a0d94d4488b627b215cf1100"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "019f5812483c6856c210c92629441794"
server_key_iv: "256ee13a540162296a7802757b859883"
server_key_version: "encryptionkey@0"
I20260926 14:50:27.974407 11734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:27.983052 11734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:27.984659 11734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:27.984808 11734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:27.988404 11734 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.189:43051
I20260926 14:50:27.988430 11801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.189:43051 every 8 connection(s)
I20260926 14:50:27.988727 11734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
I20260926 14:50:27.989600 11802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:27.990993 11802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:50:27.994199 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11734
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.189@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
Sep 26 14:50:27 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
I20260926 14:50:27.999797 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.991580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.5.15.190:38521)
0926 14:50:27.991650 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.991652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.991861 (+ 209us) server_negotiation.cc:407] Connection header received
0926 14:50:27.992288 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.992289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.992299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.992354 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.992853 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.993295 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.994203 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.994520 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.994612 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.996960 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:27.996962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:27.996962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:27.996970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:27.998409 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.998850 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.998852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.998852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.998914 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:27.999147 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:27.999148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:27.999148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:27.999310 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:27.999386 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:27.999621 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 14:50:27.999712 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:50:27.999930 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.991546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:50810)
0926 14:50:27.991707 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:27.991764 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:27.992207 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:27.992406 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:27.992409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:27.992614 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:27.992790 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.992793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.993362 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.993362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.994124 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:27.994126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.994807 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.994808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:27.994946 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:27.995452 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:27.995459 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:27.996872 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:27.998501 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.998506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.998509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.998714 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.998982 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:27.998983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:27.998983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:27.999080 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:27.999377 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:27.999382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:27.999498 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 14:50:27.999597 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 14:50:27.999727 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260926 14:50:28.000692 11802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:50:28.002761 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:27.994633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.5.15.189:43051)
0926 14:50:27.994804 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:27.994807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:27.994852 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:50:27.994915 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:27.994917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:27.994936 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:27.995005 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:27.995687 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.996497 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.997443 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:27.997719 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:27.997773 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.000087 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.000090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.000091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.000097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.001606 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.001964 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.001967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.001968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.002032 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.002224 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.002224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.002225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.002352 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.002521 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.002564 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.002633 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.189@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
I20260926 14:50:28.010273 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.000913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:60292)
0926 14:50:28.000970 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.000981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.001019 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.001258 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.001259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.001380 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.001416 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.001417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.001993 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.001994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.002434 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.002436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.002900 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.002900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.002955 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.006632 (+ 3677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.006637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.007868 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.009524 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.009525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.009525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.009675 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.009858 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.009858 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.009858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.009905 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.010058 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.010059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.010088 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.010135 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.010200 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:50:28.010396 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.000949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.5.15.189:43051)
0926 14:50:28.001152 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.001154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.001159 (+ 5us) server_negotiation.cc:407] Connection header received
0926 14:50:28.001173 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.001173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.001179 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.001254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.001491 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.001920 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.002522 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.002810 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.002863 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.007943 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.007944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.007944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.007949 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.009440 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.009749 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.009750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.009750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.009805 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.009941 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.009941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.009941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.010014 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.010054 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.010147 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.010240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260926 14:50:28.011499 11802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Bootstrap starting.
I20260926 14:50:28.012068 11802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.012331 11802 log.cc:824] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.012979 11802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: No bootstrap required, opened a new log
I20260926 14:50:28.014204 11802 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } }
I20260926 14:50:28.014329 11802 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.014374 11802 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf14746a0d94d4488b627b215cf1100, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.014511 11802 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } }
I20260926 14:50:28.014847 11802 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.014796 11809 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } } }
I20260926 14:50:28.014909 11809 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:50:28.016487 11820 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:28.016541 11820 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:28.017887 11734 master_runner.cc:428] Detected that this master cdf14746a0d94d4488b627b215cf1100 is joining an existing cluster
I20260926 14:50:28.017954 11734 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.15.189:43051
I20260926 14:50:28.025852 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.019143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.5.15.189:43051)
0926 14:50:28.019186 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.019188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.019280 (+ 92us) server_negotiation.cc:407] Connection header received
0926 14:50:28.019361 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.019362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.019370 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.019393 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.019752 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.020458 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.021342 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.021644 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.021715 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.022779 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.022781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.022781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.022787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.024679 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.025075 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.025076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.025076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.025128 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.025340 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.025341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.025342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.025463 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.025525 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.025675 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.025766 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:50:28.025962 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.018952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.5.15.190:38521)
0926 14:50:28.019008 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.019010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.019126 (+ 116us) server_negotiation.cc:407] Connection header received
0926 14:50:28.019261 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.019263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.019273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.019337 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.019693 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.020302 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.021766 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.022042 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.022115 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.023269 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.023271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.023271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.023277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.024696 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.025141 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.025143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.025143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.025218 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.025426 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.025427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.025428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.025570 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.025639 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.025791 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.025855 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:50:28.025964 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.018879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:50816)
0926 14:50:28.019065 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.019075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.019148 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.019341 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.019341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.019542 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.019611 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.019612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.020387 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.020387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.021261 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.021263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.022118 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.022119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.022183 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.022804 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.022808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.023151 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.024800 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.024801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.024802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.025071 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.025283 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.025284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.025284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.025349 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.025635 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.025636 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.025681 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.025747 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.025837 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 14:50:28.026494 11827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.019118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:60298)
0926 14:50:28.019228 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.019237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.019292 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.019450 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.019450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.019609 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.019680 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.019682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.020534 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.020534 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.021262 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.021263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.021736 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.021737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.021779 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.022334 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.022339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.022666 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.024763 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.024764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.024764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.025006 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.025184 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.025185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.025185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.025235 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.025529 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.025530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.025568 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.026298 (+ 730us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.026398 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.190@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
I20260926 14:50:28.035318 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.028154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:60306)
0926 14:50:28.028257 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.028294 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.028362 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.028490 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.028492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.028737 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.028782 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.028783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.029588 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.029588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.030498 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.030500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.030984 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.030986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.031088 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.031489 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.031493 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.032904 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.034265 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.034272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.034274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.034469 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.034676 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.034694 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.034694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.034753 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.035047 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.035053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.035093 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.035156 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.035237 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":60,"threads_started":1}
I20260926 14:50:28.035359 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.028203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.5.15.189:43051)
0926 14:50:28.028237 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.028239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.028364 (+ 125us) server_negotiation.cc:407] Connection header received
0926 14:50:28.028413 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.028413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.028424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.028487 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.028846 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.029510 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.030598 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.030914 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.030982 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.032989 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.032990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.032990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.032997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.034181 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.034541 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.034542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.034542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.034597 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.034847 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.034849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.034849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.034976 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.035050 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.035206 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.035278 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:28.035846 11270 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.15.189:43051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cdf14746a0d94d4488b627b215cf1100"
member_type: NON_VOTER
last_known_addr {
host: "127.5.15.189"
port: 43051
}
attrs {
promote: true
}
}
dest_uuid: "31006d7f535a4a9f8ac7b9faf17be594"
cas_config_opid_index: -1
I20260926 14:50:28.036175 11270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } }
I20260926 14:50:28.036654 11830 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER cdf14746a0d94d4488b627b215cf1100 (127.5.15.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } }
I20260926 14:50:28.036919 11776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:28.037684 11776 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Refusing update from remote peer 31006d7f535a4a9f8ac7b9faf17be594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 14:50:28.037703 11830 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.15.189:43051
I20260926 14:50:28.037904 11734 master.cc:537] Master@127.5.15.189:43051 shutting down...
I20260926 14:50:28.039052 11829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } }
I20260926 14:50:28.039564 11829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.039400 11830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:50:28.040217 11254 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 14:50:28.040428 11734 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:28.040515 11734 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:28.040573 11734 tablet_replica.cc:330] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: stopping tablet replica
W20260926 14:50:28.040627 11254 consensus_peers.cc:487] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 -> Peer cdf14746a0d94d4488b627b215cf1100 (127.5.15.189:43051): Couldn't send request to peer cdf14746a0d94d4488b627b215cf1100. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:28.053017 11734 master.cc:552] Master@127.5.15.189:43051 shutdown complete.
I20260926 14:50:28.053063 11734 master_runner.cc:305] Clearing existing system tablet
I20260926 14:50:28.053193 11734 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.053236 11734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.053866 11734 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:28.054346 11834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.054486 11734 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.054541 11734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.054565 11734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
uuid: "cdf14746a0d94d4488b627b215cf1100"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "019f5812483c6856c210c92629441794"
server_key_iv: "256ee13a540162296a7802757b859883"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.055311 11734 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:28.056797 11734 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:50:28.056864 11734 log.cc:1196] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 14:50:28.057047 11734 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Deleting consensus metadata
I20260926 14:50:28.057160 11734 master_runner.cc:315] Copying system tablet from 127.5.15.190:38521
I20260926 14:50:28.057741 11734 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: tablet copy: Beginning tablet copy session from remote peer at address 127.5.15.190:38521
I20260926 14:50:28.063753 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.057972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:50818)
0926 14:50:28.058087 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.058115 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.058180 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.058306 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.058307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.058527 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.058932 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.058935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.059576 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.059577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.060129 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.060130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.060495 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.060496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.060583 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.061049 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.061053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.061322 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.062716 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.062717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.062717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.062922 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.063123 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.063124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.063124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.063177 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.063435 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.063436 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.063477 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.063560 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.063655 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260926 14:50:28.063738 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.058031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.5.15.190:38521)
0926 14:50:28.058083 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.058085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.058164 (+ 79us) server_negotiation.cc:407] Connection header received
0926 14:50:28.058238 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.058238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.058249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.058293 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.059012 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.059498 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.060210 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.060412 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.060494 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.061420 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.061422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.061422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.061428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.062626 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.063000 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.063001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.063001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.063059 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.063259 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.063260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.063260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.063366 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.063433 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.063588 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.063653 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:50:28.064265 11290 tablet_copy_service.cc:140] P 31006d7f535a4a9f8ac7b9faf17be594: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cdf14746a0d94d4488b627b215cf1100 ({username='oryx', principal='oryx/127.5.15.189@KRBTEST.COM'} at 127.0.0.1:50818)
I20260926 14:50:28.064354 11290 tablet_copy_service.cc:161] P 31006d7f535a4a9f8ac7b9faf17be594: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cdf14746a0d94d4488b627b215cf1100 at {username='oryx', principal='oryx/127.5.15.189@KRBTEST.COM'} at 127.0.0.1:50818: session id = cdf14746a0d94d4488b627b215cf1100-00000000000000000000000000000000
I20260926 14:50:28.065141 11290 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:50:28.065716 11734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:50:28.066547 11734 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: tablet copy: Starting download of 0 data blocks...
I20260926 14:50:28.066702 11734 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: tablet copy: Starting download of 1 WAL segments...
I20260926 14:50:28.067566 11734 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:50:28.068262 11290 tablet_copy_service.cc:342] P 31006d7f535a4a9f8ac7b9faf17be594: Request end of tablet copy session cdf14746a0d94d4488b627b215cf1100-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.15.189@KRBTEST.COM'} at 127.0.0.1:50818
I20260926 14:50:28.068324 11290 tablet_copy_service.cc:433] P 31006d7f535a4a9f8ac7b9faf17be594: ending tablet copy session cdf14746a0d94d4488b627b215cf1100-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cdf14746a0d94d4488b627b215cf1100
I20260926 14:50:28.070650 11734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.071710 11841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.071786 11842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.071842 11844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.074903 11734 server_base.cc:1034] running on GCE node
I20260926 14:50:28.075016 11734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.075139 11734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.076309 11734 hybrid_clock.cc:648] HybridClock initialized: now 1790434228076307 us; error 32 us; skew 500 ppm
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:28.077535 11734 init.cc:377] Logged in from keytab as oryx/127.5.15.189@KRBTEST.COM (short username oryx)
I20260926 14:50:28.077796 11734 webserver.cc:460] Webserver started at http://127.5.15.189:43603/ using document root <none> and password file <none>
I20260926 14:50:28.077910 11734 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.077955 11734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.078550 11734 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:28.079115 11851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.079267 11734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.079345 11734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
uuid: "cdf14746a0d94d4488b627b215cf1100"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "019f5812483c6856c210c92629441794"
server_key_iv: "256ee13a540162296a7802757b859883"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.079396 11734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.089329 11734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.091096 11734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.093981 11734 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.189:43051
I20260926 14:50:28.094192 11903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.189:43051 every 8 connection(s)
I20260926 14:50:28.094333 11734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
I20260926 14:50:28.095067 11904 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:50:28.095451 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Bootstrap starting.
I20260926 14:50:28.097520 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:50:28.097750 11904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: Bootstrap complete.
I20260926 14:50:28.097929 11904 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } }
I20260926 14:50:28.097997 11904 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cdf14746a0d94d4488b627b215cf1100, State: Initialized, Role: LEARNER
I20260926 14:50:28.098021 11904 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } }
I20260926 14:50:28.098222 11904 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.098218 11907 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } }
I20260926 14:50:28.098332 11907 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:28.099663 11918 catalog_manager.cc:1259] Loaded cluster ID: 9b92355240654b549ff0a0d9722fbcca
I20260926 14:50:28.099706 11918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: loading cluster ID for follower catalog manager: success
I20260926 14:50:28.100708 11918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: acquiring CA information for follower catalog manager: success
I20260926 14:50:28.100966 11918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:50:28.111267 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.104003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.5.15.189:43051)
0926 14:50:28.104223 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.104225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.104233 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:28.104276 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.104277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.104287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.104345 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.105115 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.106055 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.107049 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.107366 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.107458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.108586 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.108588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.108588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.108595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.110069 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.110435 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.110436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.110437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.110493 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.110690 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.110690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.110691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.110848 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.110964 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.111075 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.111150 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.5.15.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.188.keytab.
Entry for principal oryx/127.5.15.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.188.keytab.
I20260926 14:50:28.135010 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:41809
--webserver_interface=127.5.15.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.188.keytab
--principal=oryx/127.5.15.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:28.212472 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.212677 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.212713 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.213784 11922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:28.213847 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.213871 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.213896 11922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:28.213936 11922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:28.215440 11922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.188.keytab
--principal=oryx/127.5.15.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:41809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.15.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:28.215698 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.215955 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.220510 11930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.220715 11922 server_base.cc:1034] running on GCE node
W20260926 14:50:28.220520 11928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.220523 11927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.221065 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.221292 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.222435 11922 hybrid_clock.cc:648] HybridClock initialized: now 1790434228222425 us; error 26 us; skew 500 ppm
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:28.224743 11922 init.cc:377] Logged in from keytab as oryx/127.5.15.188@KRBTEST.COM (short username oryx)
I20260926 14:50:28.225122 11922 webserver.cc:460] Webserver started at http://127.5.15.188:34983/ using document root <none> and password file <none>
I20260926 14:50:28.225334 11922 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.225376 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.225535 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:28.226504 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/instance:
uuid: "1deca22567184ee79e402e8bd31973cd"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-s4p1"
server_key: "d00d064253b457f7c50dac279bce647c"
server_key_iv: "49f3f97edf4d1aad6d8f3fdb81e65345"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.226886 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal/instance:
uuid: "1deca22567184ee79e402e8bd31973cd"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-s4p1"
server_key: "d00d064253b457f7c50dac279bce647c"
server_key_iv: "49f3f97edf4d1aad6d8f3fdb81e65345"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.228113 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:28.228773 11937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.228968 11922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:28.229050 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
uuid: "1deca22567184ee79e402e8bd31973cd"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-s4p1"
server_key: "d00d064253b457f7c50dac279bce647c"
server_key_iv: "49f3f97edf4d1aad6d8f3fdb81e65345"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.229132 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.237527 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.239293 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.239449 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.242830 11922 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.188:41809
I20260926 14:50:28.242859 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.188:41809 every 8 connection(s)
I20260926 14:50:28.243173 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
I20260926 14:50:28.244074 11990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.245402 11990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:50:28.249320 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 11922
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.188@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:28.253815 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.246054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.5.15.190:38521)
0926 14:50:28.246094 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.246096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.246176 (+ 80us) server_negotiation.cc:407] Connection header received
0926 14:50:28.246606 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.246607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.246616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.246668 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.247247 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.247722 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.248594 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.248763 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.248809 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.250872 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.250873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.250874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.250881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.252329 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.252772 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.252773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.252773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.252824 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.253111 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.253112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.253112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.253236 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.253311 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.253618 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.253713 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:28.253831 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.245904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:50832)
0926 14:50:28.246066 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.246101 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.246532 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.246720 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.246723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.246994 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.247165 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.247169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.247795 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.247796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.248520 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.248521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.248827 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.248828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.248915 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.249306 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.249316 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.250728 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.252405 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.252410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.252411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.252688 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.252887 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.252887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.252888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.253006 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.253304 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.253308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.253470 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.253577 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.253694 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260926 14:50:28.254519 11990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.15.188@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.188@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
I20260926 14:50:28.258682 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.249810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.5.15.188:41809)
0926 14:50:28.249984 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.249987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.250000 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:50:28.250016 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.250017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.250030 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.250124 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.250721 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.251534 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.252562 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.253024 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.253098 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.255574 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.255581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.255581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.255588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.257404 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.257775 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.257777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.257778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.257826 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.258089 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.258090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.258091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.258290 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.258422 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.258489 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.258561 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260926 14:50:28.261076 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.254722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:60336)
0926 14:50:28.254794 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.254803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.254856 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.255004 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.255005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.255125 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.255184 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.255186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.255814 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.255815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.256306 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.256308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.256673 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.256673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.256737 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.257089 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.257093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.258474 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.260086 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.260087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.260087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.260259 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.260463 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.260464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.260464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.260514 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.260778 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.260779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.260819 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.260906 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.260993 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 14:50:28.261090 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.254826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.5.15.189:43051)
0926 14:50:28.254887 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.254889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.254898 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:28.254923 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.254924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.254934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.255002 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.255247 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.255701 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.256389 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.256602 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.256676 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.258558 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.258559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.258560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.258567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.260005 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.260336 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.260337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.260338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.260389 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.260581 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.260582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.260582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.260712 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.260786 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.260932 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.261006 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:28.261499 11990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.188@KRBTEST.COM for oryx/127.5.15.188@KRBTEST.COM
I20260926 14:50:28.267227 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.261715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:49886)
0926 14:50:28.261762 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.261772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.261809 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.261925 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.261926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.262048 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.262068 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.262069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.262658 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.262658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.263173 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.263175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.263531 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.263532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.263579 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.263884 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.263888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.264892 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.266456 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.266457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.266457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.266596 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.266742 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.266743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.266743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.266850 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.267029 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.267030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.267060 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.267108 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.267176 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:50:28.267287 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.261783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.5.15.188:41809)
0926 14:50:28.261812 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.261814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.261821 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:28.261860 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.261861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.261869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.261922 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.262137 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.262574 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.263248 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.263451 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.263531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.264960 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.264961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.264961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.264966 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.266391 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.266651 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.266652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.266652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.266691 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.266892 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.266893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.266893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.266975 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.267022 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.267121 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.267217 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":16}
I20260926 14:50:28.268399 11990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Bootstrap starting.
I20260926 14:50:28.269209 11990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.269480 11990 log.cc:824] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.270108 11990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: No bootstrap required, opened a new log
I20260926 14:50:28.271481 11990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } }
I20260926 14:50:28.271588 11990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.271611 11990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1deca22567184ee79e402e8bd31973cd, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.271756 11990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } }
I20260926 14:50:28.272058 11990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.272030 11995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } } }
I20260926 14:50:28.272131 11995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:50:28.273597 12006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:28.273649 12006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:28.275830 11922 master_runner.cc:428] Detected that this master 1deca22567184ee79e402e8bd31973cd is joining an existing cluster
I20260926 14:50:28.275877 11922 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.15.188:41809
I20260926 14:50:28.283349 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.276903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:50846)
0926 14:50:28.277074 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.277088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.277151 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.277351 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.277354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.277478 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.277525 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.277526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.278563 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.278564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.279444 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.279446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.279923 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.279924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.279978 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.280416 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.280421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.280782 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.282349 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.282350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.282350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.282549 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.282807 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.282808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.282808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.282856 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.283108 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.283109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.283181 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.283227 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.283294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260926 14:50:28.283417 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.277106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.5.15.190:38521)
0926 14:50:28.277146 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.277148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.277157 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:28.277224 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.277225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.277236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.277292 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.277618 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.278469 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.279540 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.279822 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.279872 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.280860 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.280861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.280862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.280868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.282266 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.282630 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.282631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.282632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.282707 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.282916 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.282917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.282917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.283045 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.283114 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.283272 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.283340 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434227, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.190@KRBTEST.COM for oryx/127.5.15.188@KRBTEST.COM
I20260926 14:50:28.302564 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.285176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:49914)
0926 14:50:28.285217 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.285232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.285261 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.285492 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.285493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.285628 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.285659 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.285660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.286294 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.286296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.286864 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.286866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.287243 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.287244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.287307 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.287715 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.287719 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.288743 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.301440 (+ 12697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.301441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.301441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.301693 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.301923 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.301923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.301924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.301986 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.302251 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.302252 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.302296 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.302346 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.302440 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 14:50:28.302613 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.276935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.5.15.189:43051)
0926 14:50:28.276977 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.276979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.277322 (+ 343us) server_negotiation.cc:407] Connection header received
0926 14:50:28.277414 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.277415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.277424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.277451 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.277903 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.278629 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.279827 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.280122 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.280161 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.299878 (+ 19717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.299879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.299880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.299886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.301462 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.301878 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.301879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.301880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.301930 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.302113 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.302114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.302114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.302226 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.302265 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.302449 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.302524 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:28.302958 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.276903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:60346)
0926 14:50:28.277272 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.277282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.277351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.277510 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.277510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.277760 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.277820 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.277821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.278738 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.278739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.279736 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.279739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.280252 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.280253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.280321 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.299284 (+ 18963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.299289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.299784 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.301573 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.301574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.301574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.301800 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.301997 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.301998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.301998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.302049 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.302309 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.302309 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.302347 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.302722 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.302856 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":24,"spinlock_wait_cycles":9088,"thread_start_us":105,"threads_started":1}
I20260926 14:50:28.303261 11993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.277135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.5.15.188:41809)
0926 14:50:28.277168 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.277169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.277747 (+ 578us) server_negotiation.cc:407] Connection header received
0926 14:50:28.277818 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.277819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.277825 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.277848 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.278210 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.278878 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.279804 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.280046 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.280086 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.300121 (+ 20035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.300121 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.300122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.300127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.302172 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.302517 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.302518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.302519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.302558 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.302799 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.302800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.302801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.302915 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.302951 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.303102 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.303190 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:28.303483 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.277201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:49898)
0926 14:50:28.277659 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.277668 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.277764 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.277891 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.277891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.278069 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.278138 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.278140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.278954 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.278955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.279726 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.279727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.280148 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.280149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.280202 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.299543 (+ 19341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.299548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.300017 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.302257 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.302257 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.302258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.302448 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.302621 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.302621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.302621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.302692 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.302985 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.302986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.303015 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.303323 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.303396 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260926 14:50:28.303910 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.285225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.5.15.188:41809)
0926 14:50:28.285369 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.285370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.285375 (+ 5us) server_negotiation.cc:407] Connection header received
0926 14:50:28.285407 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.285408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.285415 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.285477 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.285739 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.286190 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.286954 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.287162 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.287247 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.288816 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.288817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.288817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.288823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.301335 (+ 12512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.301795 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.301796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.301796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.301846 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.302056 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.302057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.302058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.302177 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.303704 (+ 1527us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.303749 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.303821 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260926 14:50:28.304312 11270 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.15.188:41809: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1deca22567184ee79e402e8bd31973cd"
member_type: NON_VOTER
last_known_addr {
host: "127.5.15.188"
port: 41809
}
attrs {
promote: true
}
}
dest_uuid: "31006d7f535a4a9f8ac7b9faf17be594"
cas_config_opid_index: 5
I20260926 14:50:28.304535 11270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } }
I20260926 14:50:28.304847 11830 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1deca22567184ee79e402e8bd31973cd (127.5.15.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.305693 11830 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.15.188:41809
I20260926 14:50:28.305768 11830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.305837 11830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.305938 11922 master.cc:537] Master@127.5.15.188:41809 shutting down...
W20260926 14:50:28.306703 11254 consensus_peers.cc:487] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 -> Peer 1deca22567184ee79e402e8bd31973cd (127.5.15.188:41809): Couldn't send request to peer 1deca22567184ee79e402e8bd31973cd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:28.309856 11922 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:28.309935 11922 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:28.309975 11922 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: stopping tablet replica
I20260926 14:50:28.310109 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.306323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:60356)
0926 14:50:28.306452 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.306462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.306501 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.307334 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.307335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.307336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:28.307381 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.307382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.308064 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.308065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.309010 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.309013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.309857 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.309858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.309918 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.309941 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.309991 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.310032 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 14:50:28.310370 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.306397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.5.15.189:43051)
0926 14:50:28.307200 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.307202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.307211 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:28.307229 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.307230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.307250 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.307317 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:28.307453 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.307988 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.309097 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.309684 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.309749 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.309822 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.310270 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.310305 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785}
I20260926 14:50:28.310619 11878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 LEARNER]: Refusing update from remote peer 31006d7f535a4a9f8ac7b9faf17be594: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:50:28.310900 11829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 14:50:28.311558 11878 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1deca22567184ee79e402e8bd31973cd (127.5.15.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.312439 12017 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.312534 12017 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:28.312654 11830 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: attempting to promote NON_VOTER cdf14746a0d94d4488b627b215cf1100 to VOTER
I20260926 14:50:28.312634 11907 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31006d7f535a4a9f8ac7b9faf17be594. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: NON_VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: true } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.312726 11907 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:28.312814 11830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } }
W20260926 14:50:28.313028 11223 connection.cc:535] client connection to 127.5.15.188:41809 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:50:28.313482 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.313383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:49920)
0926 14:50:28.313420 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.313459 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:41809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 14:50:28.313535 11878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 LEARNER]: Refusing update from remote peer 31006d7f535a4a9f8ac7b9faf17be594: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 14:50:28.313820 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.313684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:49928)
0926 14:50:28.313754 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.313778 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:41809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 14:50:28.313772 12018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 14:50:28.313897 11254 consensus_peers.cc:487] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 -> Peer 1deca22567184ee79e402e8bd31973cd (127.5.15.188:41809): Couldn't send request to peer 1deca22567184ee79e402e8bd31973cd. Status: Network error: Client connection negotiation failed: client connection to 127.5.15.188:41809: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:28.314405 11829 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cdf14746a0d94d4488b627b215cf1100 (127.5.15.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.314545 11878 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, cdf14746a0d94d4488b627b215cf1100 (127.5.15.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.315294 11829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.315332 11830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.315375 11829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.315383 11830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:28.315626 11907 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.315718 11907 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:28.324321 11922 master.cc:552] Master@127.5.15.188:41809 shutdown complete.
I20260926 14:50:28.324363 11922 master_runner.cc:305] Clearing existing system tablet
I20260926 14:50:28.324483 11922 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.324525 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.325237 11922 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:28.328807 12020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.328995 11922 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.329165 11922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260926 14:50:28.329233 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
uuid: "1deca22567184ee79e402e8bd31973cd"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-s4p1"
server_key: "d00d064253b457f7c50dac279bce647c"
server_key_iv: "49f3f97edf4d1aad6d8f3fdb81e65345"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.330058 11922 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:28.331511 11922 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:50:28.331573 11922 log.cc:1196] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 14:50:28.331760 11922 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Deleting consensus metadata
I20260926 14:50:28.331933 11922 master_runner.cc:315] Copying system tablet from 127.5.15.190:38521
I20260926 14:50:28.332495 11922 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.15.190:38521
I20260926 14:50:28.339025 12026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.332722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:50848)
0926 14:50:28.332873 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.332901 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.332987 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.333177 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.333177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.333380 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.333713 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.333716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.334571 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.334572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.335149 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.335151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.335600 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.335601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.335669 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.336096 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.336100 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.336392 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.338156 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.338157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.338157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.338326 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.338507 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.338507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.338508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.338548 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.338807 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.338808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.338842 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.338897 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.338968 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260926 14:50:28.339090 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.332791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.5.15.190:38521)
0926 14:50:28.332829 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.332831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.332959 (+ 128us) server_negotiation.cc:407] Connection header received
0926 14:50:28.333069 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.333070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.333081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.333136 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.333798 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.334504 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.335231 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.335515 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.335600 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.336495 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.336496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.336497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.336504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.338079 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.338392 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.338393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.338393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.338449 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.338582 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.338583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.338583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.338695 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.338785 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.338937 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.339006 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:28.339502 11290 tablet_copy_service.cc:140] P 31006d7f535a4a9f8ac7b9faf17be594: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1deca22567184ee79e402e8bd31973cd ({username='oryx', principal='oryx/127.5.15.188@KRBTEST.COM'} at 127.0.0.1:50848)
I20260926 14:50:28.339574 11290 tablet_copy_service.cc:161] P 31006d7f535a4a9f8ac7b9faf17be594: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1deca22567184ee79e402e8bd31973cd at {username='oryx', principal='oryx/127.5.15.188@KRBTEST.COM'} at 127.0.0.1:50848: session id = 1deca22567184ee79e402e8bd31973cd-00000000000000000000000000000000
I20260926 14:50:28.340016 11290 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:50:28.340440 11922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:50:28.341260 11922 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: tablet copy: Starting download of 0 data blocks...
I20260926 14:50:28.341409 11922 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: tablet copy: Starting download of 1 WAL segments...
I20260926 14:50:28.342135 11922 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:50:28.342890 11290 tablet_copy_service.cc:342] P 31006d7f535a4a9f8ac7b9faf17be594: Request end of tablet copy session 1deca22567184ee79e402e8bd31973cd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.15.188@KRBTEST.COM'} at 127.0.0.1:50848
I20260926 14:50:28.342943 11290 tablet_copy_service.cc:433] P 31006d7f535a4a9f8ac7b9faf17be594: ending tablet copy session 1deca22567184ee79e402e8bd31973cd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1deca22567184ee79e402e8bd31973cd
I20260926 14:50:28.345359 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.346481 12030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.346598 12028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.346664 11922 server_base.cc:1034] running on GCE node
W20260926 14:50:28.346705 12027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.346948 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.347100 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.348217 11922 hybrid_clock.cc:648] HybridClock initialized: now 1790434228348207 us; error 25 us; skew 500 ppm
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:28.349593 11922 init.cc:377] Logged in from keytab as oryx/127.5.15.188@KRBTEST.COM (short username oryx)
I20260926 14:50:28.349852 11922 webserver.cc:460] Webserver started at http://127.5.15.188:36993/ using document root <none> and password file <none>
I20260926 14:50:28.349967 11922 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.350003 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.350641 11922 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:28.351218 12037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.351354 11922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.351408 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
uuid: "1deca22567184ee79e402e8bd31973cd"
format_stamp: "Formatted at 2026-09-26 14:50:28 on dist-test-slave-s4p1"
server_key: "d00d064253b457f7c50dac279bce647c"
server_key_iv: "49f3f97edf4d1aad6d8f3fdb81e65345"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.351459 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.375953 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.377636 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.380700 11922 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.188:41809
I20260926 14:50:28.380851 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.188:41809 every 8 connection(s)
I20260926 14:50:28.381047 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
I20260926 14:50:28.381752 12090 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:50:28.382153 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Bootstrap starting.
I20260926 14:50:28.384497 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:50:28.384706 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Bootstrap complete.
I20260926 14:50:28.384891 12090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } }
I20260926 14:50:28.384977 12090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1deca22567184ee79e402e8bd31973cd, State: Initialized, Role: LEARNER
I20260926 14:50:28.385026 12090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } }
I20260926 14:50:28.385236 12090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:28.385224 12093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.385317 12093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: This master's current role is: LEARNER
I20260926 14:50:28.386620 12104 catalog_manager.cc:1259] Loaded cluster ID: 9b92355240654b549ff0a0d9722fbcca
I20260926 14:50:28.386691 12104 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: loading cluster ID for follower catalog manager: success
I20260926 14:50:28.387930 12104 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: acquiring CA information for follower catalog manager: success
I20260926 14:50:28.388199 12104 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:50:28.420502 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.414009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.5.15.188:41809)
0926 14:50:28.414139 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.414140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.414147 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:28.414167 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.414168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.414176 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.414265 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.414894 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.415489 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.416255 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.416472 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.416576 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.417834 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.417835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.417836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.417843 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.419367 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.419705 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.419706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.419706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.419782 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.419986 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.419987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.419988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.420089 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.420187 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.420294 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.420390 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260926 14:50:28.421049 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11332
I20260926 14:50:28.425752 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:46617
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:38521
--webserver_port=33029
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:28.504992 12106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.505165 12106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.505187 12106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.506271 12106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.506337 12106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.506363 12106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:28.507807 12106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:46617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=33029
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:28.507974 12106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.508126 12106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:28.512966 12106 server_base.cc:1034] running on GCE node
W20260926 14:50:28.512936 12111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.512940 12112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.512933 12114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.513346 12106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.513584 12106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.514782 12106 hybrid_clock.cc:648] HybridClock initialized: now 1790434228514743 us; error 42 us; skew 500 ppm
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:28.517222 12106 init.cc:377] Logged in from keytab as oryx/127.5.15.129@KRBTEST.COM (short username oryx)
I20260926 14:50:28.517627 12106 webserver.cc:460] Webserver started at http://127.5.15.129:33029/ using document root <none> and password file <none>
I20260926 14:50:28.517855 12106 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.517935 12106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.519352 12106 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:50:28.519882 12121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.520061 12106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:28.520139 12106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "787cbee12fb34ed49903870502dcd5e0"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "f68be12d27d86ea0ccdbd275f8b7c531"
server_key_iv: "5b29b8195b15a98c3ffbaa34164d2b58"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.520440 12106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.551913 12106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.553771 12106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.553916 12106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.554131 12106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:28.554433 12106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:28.554500 12106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.554533 12106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:28.554558 12106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.560281 12106 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:46617
I20260926 14:50:28.560350 12234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:46617 every 8 connection(s)
I20260926 14:50:28.560691 12106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.188@KRBTEST.COM
I20260926 14:50:28.570937 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 12106
I20260926 14:50:28.571101 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11466
I20260926 14:50:28.572094 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.561814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:41277 (local address 127.5.15.189:43051)
0926 14:50:28.561886 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.561888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.562814 (+ 926us) server_negotiation.cc:407] Connection header received
0926 14:50:28.562933 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.562935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.562947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.563003 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.563618 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.564368 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565503 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.565714 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565771 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.568327 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.568328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.568329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.568336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.570041 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570500 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.570502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.570502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.570564 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570891 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.570892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.570892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.571051 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.571136 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.571915 (+ 779us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.571993 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:28.572376 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.561666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.5.15.129:41277)
0926 14:50:28.562729 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.562738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.562851 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.563050 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.563051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.563352 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.563512 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.563517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.564459 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.564460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.565403 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.565406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.565851 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.565911 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.566316 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.566322 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.568220 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.570119 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.570120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.570421 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.570635 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.570636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.570822 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.571126 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.571147 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.571774 (+ 627us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.571878 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.572075 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":62,"thread_start_us":55,"threads_started":1}
I20260926 14:50:28.572969 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.561614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:54459 (local address 127.5.15.188:41809)
0926 14:50:28.561656 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.561658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.561667 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:28.562665 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.562666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.562681 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.562732 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.563619 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.564359 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565528 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.565828 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565918 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.568873 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.568875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.568875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.568883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.570554 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.572182 (+ 1628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.572184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.572184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.572239 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.572437 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.572438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.572439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.572571 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.572621 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.572819 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.572910 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:28.573031 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.561375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.5.15.129:54459)
0926 14:50:28.561536 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.561577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.562547 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.562827 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.562830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.563340 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.563512 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.563517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.564459 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.564460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.565403 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.565406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.565921 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.566000 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.566490 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.566494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.568786 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.570644 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.570645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.572095 (+ 1450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.572311 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.572312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.572312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.572371 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.572652 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.572653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.572694 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.572767 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.572875 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":64,"thread_start_us":87,"threads_started":1}
I20260926 14:50:28.575516 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.561376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:58611)
0926 14:50:28.561536 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.561576 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.562561 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.562827 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.562832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.563311 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.563511 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.563515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.564211 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.564212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.565011 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.565013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.565400 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.565560 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.566151 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.566160 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.567761 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.570018 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.570026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.570421 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.570672 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.570673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.570822 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.575189 (+ 4367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.575190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.575224 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.575300 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.575403 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260926 14:50:28.575515 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.561391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:58611 (local address 127.5.15.190:38521)
0926 14:50:28.561445 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.561448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.561656 (+ 208us) server_negotiation.cc:407] Connection header received
0926 14:50:28.562666 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.562668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.562681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.562740 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.563612 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.564103 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565101 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.565306 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.565392 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.567853 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.567854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.567855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.567863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.569735 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.570519 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.570520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.570521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.570581 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.571167 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.571169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.571169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.571323 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.571386 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.575345 (+ 3959us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.575422 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:50:28.576402 12236 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38521
I20260926 14:50:28.576447 12235 heartbeater.cc:347] Connected to a master server at 127.5.15.188:41809
I20260926 14:50:28.576529 12236 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.576529 12235 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.576735 12235 heartbeater.cc:511] Master 127.5.15.188:41809 requested a full tablet report, sending...
I20260926 14:50:28.576735 12236 heartbeater.cc:511] Master 127.5.15.190:38521 requested a full tablet report, sending...
I20260926 14:50:28.577037 12237 heartbeater.cc:347] Connected to a master server at 127.5.15.189:43051
I20260926 14:50:28.577095 12237 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.577157 12237 heartbeater.cc:511] Master 127.5.15.189:43051 requested a full tablet report, sending...
I20260926 14:50:28.577227 11270 ts_manager.cc:194] Re-registered known tserver with Master: 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617)
I20260926 14:50:28.577293 12054 ts_manager.cc:194] Registered new tserver with Master: 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617)
I20260926 14:50:28.577389 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:35939
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:38521
--webserver_port=43565
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:50:28.577713 11270 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:58611
I20260926 14:50:28.579030 11868 ts_manager.cc:194] Registered new tserver with Master: 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617)
W20260926 14:50:28.658072 12242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.658264 12242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.658288 12242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.659391 12242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.659423 12242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.659448 12242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:28.660754 12242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:35939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=43565
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:28.660926 12242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.661080 12242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.665948 12247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.665957 12248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.665989 12250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.666095 12242 server_base.cc:1034] running on GCE node
I20260926 14:50:28.666350 12242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.666604 12242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.667769 12242 hybrid_clock.cc:648] HybridClock initialized: now 1790434228667767 us; error 41 us; skew 500 ppm
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:28.670017 12242 init.cc:377] Logged in from keytab as oryx/127.5.15.130@KRBTEST.COM (short username oryx)
I20260926 14:50:28.670439 12242 webserver.cc:460] Webserver started at http://127.5.15.130:43565/ using document root <none> and password file <none>
I20260926 14:50:28.670617 12242 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.670665 12242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.671988 12242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:28.672508 12257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.672663 12242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:28.672741 12242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "37ee595e906f467a990314e743cecc93"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "00f2c649c331a66b267fbcf6b62075ef"
server_key_iv: "cb59f1f4d8df0f0dc94f2a3ae564ef9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.673056 12242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.681773 12242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.683436 12242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.683531 12242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.683679 12242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:28.683936 12242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:28.683964 12242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.683981 12242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:28.684023 12242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.689826 12242 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:35939
I20260926 14:50:28.689893 12370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:35939 every 8 connection(s)
I20260926 14:50:28.690196 12242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:28.692260 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 12242
I20260926 14:50:28.692438 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11600
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.188@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:28.701162 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:37389
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:38521
--webserver_port=34049
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:41977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:50:28.702734 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.691244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:50917 (local address 127.5.15.189:43051)
0926 14:50:28.691285 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.691287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.691406 (+ 119us) server_negotiation.cc:407] Connection header received
0926 14:50:28.691737 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.691738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.691751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.691800 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.692719 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.693510 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695110 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.695451 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695490 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.699020 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.699022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.699022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.699029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.701067 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701722 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.701723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.701723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.701787 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.702064 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.702065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.702065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.702194 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.702234 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.702573 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.702648 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:28.702769 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.690898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:42565 (local address 127.5.15.188:41809)
0926 14:50:28.690958 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.690960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.691198 (+ 238us) server_negotiation.cc:407] Connection header received
0926 14:50:28.691748 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.691749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.691762 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.691793 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.692723 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.693493 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695115 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.695401 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695437 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.699379 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.699380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.699380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.699387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.701169 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701716 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.701717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.701718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.701787 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.702064 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.702065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.702065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.702180 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.702233 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.702563 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.702643 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:28.702937 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.690884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.5.15.130:42565)
0926 14:50:28.691063 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.691101 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.691631 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.691864 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.691869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.692409 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.692614 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.692620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.693641 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.693641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.695016 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.695018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.695522 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.695668 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.696880 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.696884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.699273 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.701243 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.701244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.701619 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.701853 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.701854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.701988 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.702266 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.702272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.702413 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.702523 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.702681 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":68,"thread_start_us":89,"threads_started":1}
I20260926 14:50:28.702987 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.690837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:54445 (local address 127.5.15.190:38521)
0926 14:50:28.690904 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.690906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.691198 (+ 292us) server_negotiation.cc:407] Connection header received
0926 14:50:28.691748 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.691750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.691761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.691793 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.692727 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.693489 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695115 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.695397 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695437 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.699999 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.700000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.700000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.700006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.701743 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.702201 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.702202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.702202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.702267 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.702473 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.702474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.702474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.702590 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.702641 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.702852 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.702924 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:28.703282 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.690940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:54445)
0926 14:50:28.691082 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.691101 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.691631 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.691864 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.691869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.692353 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.692614 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.692620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.693641 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.693641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.695016 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.695018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.695521 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.695668 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.696963 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.696968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.699912 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.701909 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.701910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.702122 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.702341 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.702342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.702342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.702402 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.702673 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.702674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.702709 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.703076 (+ 367us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.703177 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":134,"spinlock_wait_cycles":8064,"thread_start_us":60,"threads_started":1}
I20260926 14:50:28.703353 12371 heartbeater.cc:347] Connected to a master server at 127.5.15.188:41809
I20260926 14:50:28.703420 12371 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.703547 12371 heartbeater.cc:511] Master 127.5.15.188:41809 requested a full tablet report, sending...
I20260926 14:50:28.703791 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.691191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.5.15.130:50917)
0926 14:50:28.691333 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.691342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.691647 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.691864 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.691867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.692397 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.692615 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.692621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.693646 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.693646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.695012 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.695015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.695561 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.695562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.695668 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.696812 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.696822 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.698916 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.701188 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.701195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.701643 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.701851 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.701852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.701852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.701988 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.702280 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.702280 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.702413 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.703609 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.703702 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":122,"thread_start_us":67,"threads_started":1}
I20260926 14:50:28.703944 12054 ts_manager.cc:194] Registered new tserver with Master: 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:28.704094 12372 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38521
I20260926 14:50:28.704140 12372 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.704216 12372 heartbeater.cc:511] Master 127.5.15.190:38521 requested a full tablet report, sending...
I20260926 14:50:28.704406 11270 ts_manager.cc:194] Re-registered known tserver with Master: 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:28.704468 12373 heartbeater.cc:347] Connected to a master server at 127.5.15.189:43051
I20260926 14:50:28.704523 12373 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.704589 12373 heartbeater.cc:511] Master 127.5.15.189:43051 requested a full tablet report, sending...
I20260926 14:50:28.704856 11270 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:54445
I20260926 14:50:28.704859 11868 ts_manager.cc:194] Registered new tserver with Master: 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
W20260926 14:50:28.781951 12378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:28.782132 12378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:28.782153 12378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:28.783244 12378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:28.783281 12378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:28.783306 12378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:28.784701 12378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:41977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:37389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=34049
--tserver_master_addrs=127.5.15.190:38521,127.5.15.189:43051,127.5.15.188:41809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:28.784897 12378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:28.785054 12378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:28.789922 12384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.789968 12383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:28.789973 12386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:28.790091 12378 server_base.cc:1034] running on GCE node
I20260926 14:50:28.790328 12378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:28.790582 12378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:28.791733 12378 hybrid_clock.cc:648] HybridClock initialized: now 1790434228791715 us; error 48 us; skew 500 ppm
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:28.794055 12378 init.cc:377] Logged in from keytab as oryx/127.5.15.131@KRBTEST.COM (short username oryx)
I20260926 14:50:28.794446 12378 webserver.cc:460] Webserver started at http://127.5.15.131:34049/ using document root <none> and password file <none>
I20260926 14:50:28.794663 12378 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:28.794744 12378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:28.795964 12378 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:28.796494 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.796651 12378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:28.796746 12378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45"
format_stamp: "Formatted at 2026-09-26 14:50:27 on dist-test-slave-s4p1"
server_key: "29017a39698c6cc7e2a6e634a6a5d034"
server_key_iv: "e9575748f43f0dcca5ec900b9f2a3a03"
server_key_version: "encryptionkey@0"
I20260926 14:50:28.797096 12378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:28.810439 12378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:28.812206 12378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:28.812358 12378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:28.812577 12378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:28.812882 12378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:28.812934 12378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.812968 12378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:28.812994 12378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:28.819166 12378 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:37389
I20260926 14:50:28.819232 12506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:37389 every 8 connection(s)
I20260926 14:50:28.819572 12378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.188@KRBTEST.COM
Sep 26 14:50:28 dist-test-slave-s4p1 krb5kdc[11225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434228, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.189@KRBTEST.COM
I20260926 14:50:28.827389 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 12378
I20260926 14:50:28.829885 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.820251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:55403 (local address 127.5.15.190:38521)
0926 14:50:28.820288 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.820290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.820476 (+ 186us) server_negotiation.cc:407] Connection header received
0926 14:50:28.821116 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.821117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.821129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.821189 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.821991 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.822671 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.823822 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.824119 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.824184 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.826370 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.826371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.826372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.826378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.828332 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.828829 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.828830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.828830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.828892 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.829149 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.829149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.829150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.829264 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.829337 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.829698 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.829779 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:50:28.830379 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.820205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:55403)
0926 14:50:28.820359 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.820398 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.821006 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.821253 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.821258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.821704 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.821895 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.821900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.822780 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.822781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.823717 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.823718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.824237 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.824238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.824314 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.824703 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.824707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.826279 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.828420 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.828426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.828428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.828732 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.828955 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.828956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.828956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.829072 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.829329 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.829335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.829524 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.829995 (+ 471us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.830155 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":53,"thread_start_us":85,"threads_started":1}
I20260926 14:50:28.830432 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.820214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:43127 (local address 127.5.15.188:41809)
0926 14:50:28.820258 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.820261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.820622 (+ 361us) server_negotiation.cc:407] Connection header received
0926 14:50:28.821104 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.821106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.821117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.821176 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.821991 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.822437 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.823647 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.823885 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.823960 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.827176 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.827177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.827177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.827184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.828825 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.829224 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.829226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.829226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.829290 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.829519 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.829521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.829521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.829687 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.829758 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.830275 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.830339 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:28.830614 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.820205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.5.15.131:43127)
0926 14:50:28.820557 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.820568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.821017 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.821252 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.821256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.821703 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.821894 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.821898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.822532 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.822533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.823526 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.823528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.823960 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.823961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.824110 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.824593 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.824600 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.827051 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.828913 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.828915 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.828915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.829130 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.829359 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.829360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.829360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.829417 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.830034 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.830035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.830074 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.830143 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.830208 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 14:50:28.830772 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.820634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:53299 (local address 127.5.15.189:43051)
0926 14:50:28.820679 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.820681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.821229 (+ 548us) server_negotiation.cc:407] Connection header received
0926 14:50:28.821285 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.821286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.821298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.821353 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.821990 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.822675 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.823794 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.824094 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.824172 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.827460 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.827461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.827462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.827469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.829546 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.829940 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.829941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.829941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.830003 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.830192 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.830193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.830193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.830321 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.830363 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.830575 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.830664 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:28.831012 12507 heartbeater.cc:347] Connected to a master server at 127.5.15.188:41809
I20260926 14:50:28.830993 12508 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38521
I20260926 14:50:28.831105 12508 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.831104 12507 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.831205 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.820464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.5.15.131:53299)
0926 14:50:28.821180 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.821189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.821238 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.821364 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.821365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.821704 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:28.821895 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.821900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.822780 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.822781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.823716 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.823718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.824194 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.824196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.824263 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.824985 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:28.824989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:28.827360 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:28.829634 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.829634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.829635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.829855 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.830078 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:28.830079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:28.830079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:28.830128 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:28.830411 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:28.830412 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:28.830454 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.830875 (+ 421us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.830976 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":24,"thread_start_us":61,"threads_started":1}
I20260926 14:50:28.831266 12508 heartbeater.cc:511] Master 127.5.15.190:38521 requested a full tablet report, sending...
I20260926 14:50:28.831267 12507 heartbeater.cc:511] Master 127.5.15.188:41809 requested a full tablet report, sending...
I20260926 14:50:28.831525 12509 heartbeater.cc:347] Connected to a master server at 127.5.15.189:43051
I20260926 14:50:28.831589 12509 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:28.831647 12509 heartbeater.cc:511] Master 127.5.15.189:43051 requested a full tablet report, sending...
I20260926 14:50:28.831671 12054 ts_manager.cc:194] Registered new tserver with Master: 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389)
I20260926 14:50:28.831808 11270 ts_manager.cc:194] Re-registered known tserver with Master: 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389)
I20260926 14:50:28.831926 11868 ts_manager.cc:194] Registered new tserver with Master: 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389)
I20260926 14:50:28.832304 11270 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:55403
I20260926 14:50:28.839740 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.830809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.5.15.188:41809)
0926 14:50:28.830989 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.830991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.831993 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 14:50:28.832086 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.832087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.832098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.832147 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.832876 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.833357 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.834014 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.834205 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.834270 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.836451 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.836452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.836452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.836459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.838231 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.838597 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.838598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.838599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.838650 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.838900 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.838901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.838901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.839020 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.839109 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.839564 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.839647 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260926 14:50:28.839803 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.829936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.5.15.189:43051)
0926 14:50:28.830831 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.830832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.831827 (+ 995us) server_negotiation.cc:407] Connection header received
0926 14:50:28.831921 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.831922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.831931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.831984 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.832897 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.833585 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.834548 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.834876 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.834914 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.836566 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.836567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.836567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.836574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.838658 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.839066 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.839067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.839067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.839119 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.839320 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.839320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.839321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.839435 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.839523 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.839654 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.839713 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":67,"threads_started":1}
I20260926 14:50:28.840371 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.829843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.5.15.190:38521)
0926 14:50:28.829968 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.829970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.832679 (+ 2709us) server_negotiation.cc:407] Connection header received
0926 14:50:28.832769 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.832770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.832781 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.832810 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:28.833481 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.834146 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.835189 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.835462 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.835506 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.836968 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:28.836970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:28.836970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:28.836977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:28.838435 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.838828 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.838829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.838830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.838881 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:28.839096 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:28.839097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:28.839098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:28.839231 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:28.839360 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.840189 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.840246 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
I20260926 14:50:28.841713 11270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:28.843319 11270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:28.850268 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.846659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46617 (local address 127.0.0.1:33062)
0926 14:50:28.846822 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.846832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.846875 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.847154 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.847155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.847157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:28.847197 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.847198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.848205 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.848206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.849093 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.849095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.850039 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.850040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.850099 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.850103 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.850165 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.850202 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260926 14:50:28.850308 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.846777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.5.15.129:46617)
0926 14:50:28.846971 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.846975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.846984 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:28.847028 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.847030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.847043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.847155 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:28.847289 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.848124 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.849190 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.849959 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.850027 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.850091 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.850177 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.850215 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260926 14:50:28.850730 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.846892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35939 (local address 127.0.0.1:35052)
0926 14:50:28.846989 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.846998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.847036 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.847637 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.847637 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.847638 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:28.847663 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.847664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.848763 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.848764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.849623 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.849625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.850517 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.850518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.850575 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.850580 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.850628 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.850670 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":67,"threads_started":1}
I20260926 14:50:28.850850 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.847130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.5.15.131:37389)
0926 14:50:28.847282 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.847285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.847294 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:28.847326 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.847328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.847341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.847411 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:28.847587 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.848570 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.849751 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.850492 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.850585 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.850653 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.850724 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.850785 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260926 14:50:28.850865 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.846700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37389 (local address 127.0.0.1:44560)
0926 14:50:28.846835 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.846846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.846895 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.847460 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.847461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.847463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:28.847509 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.847510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.848655 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.848655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.849659 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.849661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.850589 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.850590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.850652 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.850656 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.850712 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.850799 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":94,"threads_started":1}
I20260926 14:50:28.851011 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.847293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.5.15.130:35939)
0926 14:50:28.847485 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.847489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.847496 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:28.847524 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.847525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.847536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.847641 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:28.847716 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.848690 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.849711 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.850366 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.850434 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.850486 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.850899 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.850939 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260926 14:50:28.851287 12169 tablet_service.cc:1467] Processing CreateTablet for tablet bd7913bac7904efda99a1f5faf567665 (DEFAULT_TABLE table=test-table [id=800d5c40cbbd437095e943bbc201046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:28.851450 12441 tablet_service.cc:1467] Processing CreateTablet for tablet bd7913bac7904efda99a1f5faf567665 (DEFAULT_TABLE table=test-table [id=800d5c40cbbd437095e943bbc201046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:28.851629 12169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7913bac7904efda99a1f5faf567665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.851686 12441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7913bac7904efda99a1f5faf567665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.851613 12305 tablet_service.cc:1467] Processing CreateTablet for tablet bd7913bac7904efda99a1f5faf567665 (DEFAULT_TABLE table=test-table [id=800d5c40cbbd437095e943bbc201046c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:28.851898 12305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd7913bac7904efda99a1f5faf567665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:28.854152 12534 tablet_bootstrap.cc:492] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Bootstrap starting.
I20260926 14:50:28.854677 12534 tablet_bootstrap.cc:654] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.854959 12535 tablet_bootstrap.cc:492] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Bootstrap starting.
I20260926 14:50:28.855011 12534 log.cc:824] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.855533 12535 tablet_bootstrap.cc:654] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.855537 12536 tablet_bootstrap.cc:492] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Bootstrap starting.
I20260926 14:50:28.855751 12534 tablet_bootstrap.cc:492] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: No bootstrap required, opened a new log
I20260926 14:50:28.855815 12534 ts_tablet_manager.cc:1397] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:28.855850 12535 log.cc:824] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.856132 12536 tablet_bootstrap.cc:654] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:28.856467 12536 log.cc:824] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:28.856637 12535 tablet_bootstrap.cc:492] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: No bootstrap required, opened a new log
I20260926 14:50:28.856704 12535 ts_tablet_manager.cc:1397] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:28.857250 12536 tablet_bootstrap.cc:492] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: No bootstrap required, opened a new log
I20260926 14:50:28.857318 12536 ts_tablet_manager.cc:1397] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:28.858225 12534 raft_consensus.cc:348] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:28.858363 12534 raft_consensus.cc:374] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.858404 12534 raft_consensus.cc:729] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce4cabce51e448fbc67b1f8e7a0cc45, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.858515 12534 consensus_queue.cc:260] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:28.858824 12534 ts_tablet_manager.cc:1428] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:28.858959 12508 heartbeater.cc:503] Master 127.5.15.190:38521 was elected leader, sending a full tablet report...
I20260926 14:50:28.859439 12535 raft_consensus.cc:348] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:28.859575 12535 raft_consensus.cc:374] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.859606 12535 raft_consensus.cc:729] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cbee12fb34ed49903870502dcd5e0, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.859735 12535 consensus_queue.cc:260] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:28.859987 12535 ts_tablet_manager.cc:1428] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:28.860343 12236 heartbeater.cc:503] Master 127.5.15.190:38521 was elected leader, sending a full tablet report...
I20260926 14:50:28.860337 12536 raft_consensus.cc:348] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:28.860484 12536 raft_consensus.cc:374] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:28.860527 12536 raft_consensus.cc:729] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ee595e906f467a990314e743cecc93, State: Initialized, Role: FOLLOWER
I20260926 14:50:28.860625 12536 consensus_queue.cc:260] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:28.860912 12372 heartbeater.cc:503] Master 127.5.15.190:38521 was elected leader, sending a full tablet report...
I20260926 14:50:28.860903 12536 ts_tablet_manager.cc:1428] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 14:50:28.900933 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.898353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.5.15.188:41809)
0926 14:50:28.898403 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:28.898405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:28.898419 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:50:28.898440 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:28.898440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:28.898458 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:28.898515 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:28.898626 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.899268 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.900231 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.900668 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.900723 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.900797 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:28.900845 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:50:28.900868 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:50:28.900944 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:28.898278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:49958)
0926 14:50:28.898332 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:28.898343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:28.898387 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:28.898526 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:28.898527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:28.898528 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:28.898572 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.898573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.899364 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.899365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.900164 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:28.900166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:28.900736 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:28.900737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:28.900796 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:28.900799 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:28.900842 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:50:28.900877 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:50:28.901386 12064 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 14:50:28.902256 12543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31006d7f535a4a9f8ac7b9faf17be594. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:28.902371 12543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: This master's current role is: LEARNER
W20260926 14:50:28.940987 12374 tablet.cc:2369] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:29.061740 12238 tablet.cc:2369] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:29.070860 12510 tablet.cc:2369] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:29.112380 12540 raft_consensus.cc:482] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:29.112476 12540 raft_consensus.cc:504] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:29.112847 12540 leader_election.cc:288] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617), 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:29.115775 12512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.113028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46617 (local address 127.5.15.131:52861)
0926 14:50:29.113087 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:29.113102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:29.113163 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:29.113306 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:29.113307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:29.113308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:29.113380 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:29.113383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.113999 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.114001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:29.114875 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:29.114877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.115512 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.115513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:29.115614 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.115619 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:29.115665 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:50:29.115704 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:50:29.115803 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.113155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:52861 (local address 127.5.15.129:46617)
0926 14:50:29.113197 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:29.113199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:29.113207 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:29.113235 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:29.113236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:29.113245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:29.113302 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:29.113436 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.113906 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.114989 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.115439 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.115493 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.115500 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:29.115684 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:50:29.115723 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:29.115952 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.113028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35939 (local address 127.5.15.131:56565)
0926 14:50:29.113107 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:29.113116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:29.113168 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:29.113543 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:29.113543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:29.113544 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:29.113607 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:29.113609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.114192 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.114193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:29.115085 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:29.115087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.115676 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.115677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:29.115750 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.115754 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:29.115807 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:50:29.115865 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":40}
I20260926 14:50:29.115968 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.113368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:56565 (local address 127.5.15.130:35939)
0926 14:50:29.113426 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:29.113428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:29.113435 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:29.113456 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:29.113457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:29.113467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:29.113528 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:29.113677 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.114106 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.115165 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.115563 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.115662 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.115671 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:29.115825 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 14:50:29.115891 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:50:29.116273 12325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7913bac7904efda99a1f5faf567665" candidate_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ee595e906f467a990314e743cecc93" is_pre_election: true
I20260926 14:50:29.116417 12325 raft_consensus.cc:2454] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce4cabce51e448fbc67b1f8e7a0cc45 in term 0.
I20260926 14:50:29.116516 12189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7913bac7904efda99a1f5faf567665" candidate_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cbee12fb34ed49903870502dcd5e0" is_pre_election: true
I20260926 14:50:29.116691 12189 raft_consensus.cc:2454] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce4cabce51e448fbc67b1f8e7a0cc45 in term 0.
I20260926 14:50:29.116703 12397 leader_election.cc:302] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ee595e906f467a990314e743cecc93, 5ce4cabce51e448fbc67b1f8e7a0cc45; no voters:
I20260926 14:50:29.116830 12540 raft_consensus.cc:2781] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:29.116881 12540 raft_consensus.cc:482] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:29.116914 12540 raft_consensus.cc:3037] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:29.117583 12540 raft_consensus.cc:504] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:29.117722 12540 leader_election.cc:288] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [CANDIDATE]: Term 1 election: Requested vote from peers 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617), 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:29.117866 12325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7913bac7904efda99a1f5faf567665" candidate_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ee595e906f467a990314e743cecc93"
I20260926 14:50:29.117874 12189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd7913bac7904efda99a1f5faf567665" candidate_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787cbee12fb34ed49903870502dcd5e0"
I20260926 14:50:29.117951 12325 raft_consensus.cc:3037] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:29.117969 12189 raft_consensus.cc:3037] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:29.118652 12325 raft_consensus.cc:2454] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce4cabce51e448fbc67b1f8e7a0cc45 in term 1.
I20260926 14:50:29.118731 12189 raft_consensus.cc:2454] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce4cabce51e448fbc67b1f8e7a0cc45 in term 1.
I20260926 14:50:29.118820 12397 leader_election.cc:302] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ee595e906f467a990314e743cecc93, 5ce4cabce51e448fbc67b1f8e7a0cc45; no voters:
I20260926 14:50:29.118935 12540 raft_consensus.cc:2781] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:29.119071 12540 raft_consensus.cc:686] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce4cabce51e448fbc67b1f8e7a0cc45, State: Running, Role: LEADER
I20260926 14:50:29.119187 12540 consensus_queue.cc:237] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:29.119944 11270 catalog_manager.cc:5495] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:29.120568 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: NON_VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: true } } }
I20260926 14:50:29.120656 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:29.121284 12550 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: attempting to promote NON_VOTER 1deca22567184ee79e402e8bd31973cd to VOTER
I20260926 14:50:29.121420 12550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:29.121817 11878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Refusing update from remote peer 31006d7f535a4a9f8ac7b9faf17be594: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 14:50:29.121943 12064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 LEARNER]: Refusing update from remote peer 31006d7f535a4a9f8ac7b9faf17be594: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 14:50:29.122141 12550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 14:50:29.122262 11830 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 14:50:29.122982 12524 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 1deca22567184ee79e402e8bd31973cd (127.5.15.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.123136 11878 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 1deca22567184ee79e402e8bd31973cd (127.5.15.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.123107 12064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 1deca22567184ee79e402e8bd31973cd (127.5.15.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.123864 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.123951 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:29.124004 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.124064 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:29.124040 12543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.124107 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.124127 12543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:29.124152 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:29.124155 12553 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "31006d7f535a4a9f8ac7b9faf17be594" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:29.124243 12553 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:29.133981 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.131849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.5.15.131:37389)
0926 14:50:29.131890 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:29.131892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:29.131899 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:29.131968 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:29.131969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:29.131992 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:29.132059 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:29.132179 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.132601 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.133302 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.133482 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.133552 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.133715 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:29.133800 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:29.133884 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:29.133900 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 14:50:29.133917 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:50:29.135486 12189 raft_consensus.cc:1261] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Refusing update from remote peer 5ce4cabce51e448fbc67b1f8e7a0cc45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:29.135486 12325 raft_consensus.cc:1261] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Refusing update from remote peer 5ce4cabce51e448fbc67b1f8e7a0cc45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:29.135810 12540 consensus_queue.cc:1035] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:29.135919 12548 consensus_queue.cc:1035] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:29.140242 12557 mvcc.cc:205] Tried to move back new op lower bound from 7333618602536833024 to 7333618602472734720. Current Snapshot: MvccSnapshot[applied={T|T < 7333618602536833024 or (T in {7333618602536833024})}]
I20260926 14:50:29.140820 11269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:29.140902 11269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:29.142064 11269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31006d7f535a4a9f8ac7b9faf17be594: Sending DeleteTablet for 3 replicas of tablet bd7913bac7904efda99a1f5faf567665
I20260926 14:50:29.142346 12169 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7913bac7904efda99a1f5faf567665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:29 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:33062
I20260926 14:50:29.142475 12441 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7913bac7904efda99a1f5faf567665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:29 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:44560
I20260926 14:50:29.142693 12565 tablet_replica.cc:330] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: stopping tablet replica
I20260926 14:50:29.142850 12565 raft_consensus.cc:2229] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:29.142992 12565 raft_consensus.cc:2258] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:29.143105 12305 tablet_service.cc:1514] Processing DeleteTablet for tablet bd7913bac7904efda99a1f5faf567665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:29 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:35052
I20260926 14:50:29.143337 12566 tablet_replica.cc:330] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: stopping tablet replica
I20260926 14:50:29.143436 12566 raft_consensus.cc:2229] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:29.143530 12566 raft_consensus.cc:2258] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:29.143613 12565 ts_tablet_manager.cc:1905] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:29.143743 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11238
I20260926 14:50:29.143898 12566 ts_tablet_manager.cc:1905] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:29.145560 12566 ts_tablet_manager.cc:1918] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:29.145635 12566 log.cc:1196] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/bd7913bac7904efda99a1f5faf567665
I20260926 14:50:29.145908 12566 ts_tablet_manager.cc:1939] T bd7913bac7904efda99a1f5faf567665 P 37ee595e906f467a990314e743cecc93: Deleting consensus metadata
I20260926 14:50:29.146157 12565 ts_tablet_manager.cc:1918] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:29.146270 12565 log.cc:1196] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/bd7913bac7904efda99a1f5faf567665
I20260926 14:50:29.146595 12565 ts_tablet_manager.cc:1939] T bd7913bac7904efda99a1f5faf567665 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Deleting consensus metadata
W20260926 14:50:29.149825 12258 connection.cc:535] server connection from 127.0.0.1:35052 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 14:50:29.149911 12395 connection.cc:535] server connection from 127.0.0.1:44560 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:50:29.150466 12564 tablet_replica.cc:330] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: stopping tablet replica
I20260926 14:50:29.150579 12564 raft_consensus.cc:2229] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:29.150678 12564 raft_consensus.cc:2258] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:29.151139 12564 ts_tablet_manager.cc:1905] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:29.152904 12564 ts_tablet_manager.cc:1918] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:29.152969 12564 log.cc:1196] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/bd7913bac7904efda99a1f5faf567665
I20260926 14:50:29.153242 12564 ts_tablet_manager.cc:1939] T bd7913bac7904efda99a1f5faf567665 P 787cbee12fb34ed49903870502dcd5e0: Deleting consensus metadata
W20260926 14:50:29.153518 12124 connection.cc:441] server connection from 127.0.0.1:33062 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33062 (request call id 1) could send its response
I20260926 14:50:29.159162 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.151763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.5.15.188:41809)
0926 14:50:29.151803 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:29.151804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:29.151942 (+ 138us) server_negotiation.cc:407] Connection header received
0926 14:50:29.152045 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:29.152046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:29.152056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:29.152087 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:29.152772 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.153502 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.154315 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.154597 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.154659 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.156092 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:29.156094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:29.156094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:29.156100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:29.157883 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:29.158295 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:29.158297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:29.158297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:29.158362 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:29.158563 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:29.158564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:29.158565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:29.158690 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:29.158809 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:29.158980 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:50:29.159058 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:29.159370 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:29.151394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.5.15.189:43051)
0926 14:50:29.151445 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:29.151447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:29.151621 (+ 174us) server_negotiation.cc:407] Connection header received
0926 14:50:29.151725 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:29.151725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:29.151735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:29.151836 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:29.152557 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.153311 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.154404 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:29.154685 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:29.154776 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:29.156030 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:29.156031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:29.156031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:29.156038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:29.158086 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:29.158492 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:29.158493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:29.158494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:29.158544 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:29.158780 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:29.158781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:29.158782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:29.158903 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:29.159000 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:29.159205 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 14:50:29.159286 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:30.139940 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.139675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:43723)
0926 14:50:30.139807 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.139879 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260926 14:50:30.140255 12372 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:38521 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.141265 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.141011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:53919)
0926 14:50:30.141156 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.141222 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260926 14:50:30.141319 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.141098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:50293)
0926 14:50:30.141207 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.141269 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
W20260926 14:50:30.141484 12236 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:38521 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
W20260926 14:50:30.141724 12508 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:38521 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.643150 12576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31006d7f535a4a9f8ac7b9faf17be594)
I20260926 14:50:30.643242 12577 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31006d7f535a4a9f8ac7b9faf17be594)
I20260926 14:50:30.643337 12576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:30.643369 12577 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:30.643631 12577 leader_election.cc:288] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521), 1deca22567184ee79e402e8bd31973cd (127.5.15.188:41809)
I20260926 14:50:30.643821 12576 leader_election.cc:288] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521), cdf14746a0d94d4488b627b215cf1100 (127.5.15.189:43051)
I20260926 14:50:30.643966 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.643745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44334)
0926 14:50:30.643851 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.643904 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":44,"threads_started":1}
W20260926 14:50:30.644073 12040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.644186 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.643940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44350)
0926 14:50:30.644098 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.644151 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":53,"threads_started":1}
W20260926 14:50:30.644345 11854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.644482 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.644402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44364)
0926 14:50:30.644432 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.644455 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 14:50:30.644593 12040 leader_election.cc:334] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.644822 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.644710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44378)
0926 14:50:30.644749 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.644781 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 14:50:30.644897 11854 leader_election.cc:334] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.647734 12580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.643940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.5.15.188:41809)
0926 14:50:30.644115 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:30.644117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:30.644321 (+ 204us) server_negotiation.cc:407] Connection header received
0926 14:50:30.644480 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:30.644481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:30.644489 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:30.644579 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:30.644836 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.645510 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.646832 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.647374 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.647448 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.647459 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:30.647603 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:50:30.647643 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":40,"threads_started":1}
I20260926 14:50:30.647737 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.644016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:41809 (local address 127.0.0.1:47756)
0926 14:50:30.644251 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.644260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:30.644323 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:30.644704 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:30.644705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:30.644708 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:30.644759 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.644760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.645589 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.645589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.646728 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.646730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.647457 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.647458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.647528 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.647532 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:30.647586 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:50:30.647628 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":45,"threads_started":1}
I20260926 14:50:30.647738 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.643767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:43051 (local address 127.0.0.1:46130)
0926 14:50:30.643935 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.643946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:30.644000 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:30.644345 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:30.644346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:30.644348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:30.644424 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.644426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.645501 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.645502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.646728 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:30.646730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.647460 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.647461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:30.647523 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.647528 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:30.647586 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:50:30.647638 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":32,"threads_started":1}
I20260926 14:50:30.647748 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.644017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.5.15.189:43051)
0926 14:50:30.644169 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:30.644172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:30.644180 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:30.644208 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:30.644208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:30.644219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:30.644622 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:30.644660 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.645424 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.646818 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:30.647374 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:30.647448 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:30.647459 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:30.647603 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:50:30.647643 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":36,"threads_started":1}
I20260926 14:50:30.648013 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdf14746a0d94d4488b627b215cf1100" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1deca22567184ee79e402e8bd31973cd" is_pre_election: true
I20260926 14:50:30.648121 12064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdf14746a0d94d4488b627b215cf1100 in term 1.
I20260926 14:50:30.648224 11878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1deca22567184ee79e402e8bd31973cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cdf14746a0d94d4488b627b215cf1100" is_pre_election: true
I20260926 14:50:30.648255 11852 leader_election.cc:302] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1deca22567184ee79e402e8bd31973cd, cdf14746a0d94d4488b627b215cf1100; no voters: 31006d7f535a4a9f8ac7b9faf17be594
I20260926 14:50:30.648341 11878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1deca22567184ee79e402e8bd31973cd in term 1.
I20260926 14:50:30.648386 12577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:50:30.648418 12577 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31006d7f535a4a9f8ac7b9faf17be594)
I20260926 14:50:30.648447 12577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:50:30.648494 12038 leader_election.cc:302] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1deca22567184ee79e402e8bd31973cd, cdf14746a0d94d4488b627b215cf1100; no voters: 31006d7f535a4a9f8ac7b9faf17be594
I20260926 14:50:30.648594 12576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:50:30.648658 12576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31006d7f535a4a9f8ac7b9faf17be594)
I20260926 14:50:30.648710 12576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:50:30.649189 12577 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:30.649330 12577 leader_election.cc:288] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [CANDIDATE]: Term 2 election: Requested vote from peers 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521), 1deca22567184ee79e402e8bd31973cd (127.5.15.188:41809)
I20260926 14:50:30.649356 12576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:30.649475 12576 leader_election.cc:288] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 2 election: Requested vote from peers 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521), cdf14746a0d94d4488b627b215cf1100 (127.5.15.189:43051)
I20260926 14:50:30.649485 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.649396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44388)
0926 14:50:30.649436 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.649459 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 14:50:30.649536 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdf14746a0d94d4488b627b215cf1100" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1deca22567184ee79e402e8bd31973cd"
I20260926 14:50:30.649619 12064 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cdf14746a0d94d4488b627b215cf1100 in current term 2: Already voted for candidate 1deca22567184ee79e402e8bd31973cd in this term.
I20260926 14:50:30.649600 11878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1deca22567184ee79e402e8bd31973cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cdf14746a0d94d4488b627b215cf1100"
I20260926 14:50:30.649672 11878 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1deca22567184ee79e402e8bd31973cd in current term 2: Already voted for candidate cdf14746a0d94d4488b627b215cf1100 in this term.
I20260926 14:50:30.649801 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.649720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44402)
0926 14:50:30.649758 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.649780 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:50:30.649878 11854 leader_election.cc:334] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.649893 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.649527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44392)
0926 14:50:30.649808 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.649865 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260926 14:50:30.649919 11854 leader_election.cc:302] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdf14746a0d94d4488b627b215cf1100; no voters: 1deca22567184ee79e402e8bd31973cd, 31006d7f535a4a9f8ac7b9faf17be594
I20260926 14:50:30.650033 12577 raft_consensus.cc:2726] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 14:50:30.650200 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:30.650113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44408)
0926 14:50:30.650143 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:30.650169 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 14:50:30.650266 12040 leader_election.cc:334] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:30.650332 12040 leader_election.cc:302] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1deca22567184ee79e402e8bd31973cd; no voters: 31006d7f535a4a9f8ac7b9faf17be594, cdf14746a0d94d4488b627b215cf1100
I20260926 14:50:30.650390 12576 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 14:50:31.141096 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.140806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:51323)
0926 14:50:31.140991 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:31.141040 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260926 14:50:31.141991 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.141786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:60341)
0926 14:50:31.141908 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:31.141949 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260926 14:50:31.142232 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:31.141999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:50491)
0926 14:50:31.142157 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:31.142193 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260926 14:50:32.141867 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.141656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:51507)
0926 14:50:32.141774 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.141812 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260926 14:50:32.142603 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.142422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:48649)
0926 14:50:32.142518 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.142558 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
I20260926 14:50:32.142802 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.142598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:59791)
0926 14:50:32.142717 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.142763 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260926 14:50:32.173995 12592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:32.174098 12592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:32.174297 12592 leader_election.cc:288] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521), cdf14746a0d94d4488b627b215cf1100 (127.5.15.189:43051)
I20260926 14:50:32.174525 11878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1deca22567184ee79e402e8bd31973cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cdf14746a0d94d4488b627b215cf1100" is_pre_election: true
I20260926 14:50:32.174597 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.174424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44610)
0926 14:50:32.174531 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.174561 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260926 14:50:32.174623 11878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1deca22567184ee79e402e8bd31973cd in term 2.
I20260926 14:50:32.174788 12038 leader_election.cc:302] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1deca22567184ee79e402e8bd31973cd, cdf14746a0d94d4488b627b215cf1100; no voters:
I20260926 14:50:32.174898 12592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 14:50:32.174962 12592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:32.174988 12592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:50:32.175096 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.174971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44622)
0926 14:50:32.175020 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.175051 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:50:32.175189 12040 leader_election.cc:334] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:32.175839 12592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:32.175966 12592 leader_election.cc:288] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 3 election: Requested vote from peers 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521), cdf14746a0d94d4488b627b215cf1100 (127.5.15.189:43051)
I20260926 14:50:32.176124 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.176029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44632)
0926 14:50:32.176066 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.176103 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 14:50:32.176237 11878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1deca22567184ee79e402e8bd31973cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cdf14746a0d94d4488b627b215cf1100"
I20260926 14:50:32.176350 11878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:50:32.176560 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.176473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44634)
0926 14:50:32.176502 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.176529 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 14:50:32.176656 12040 leader_election.cc:334] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
I20260926 14:50:32.177085 11878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1deca22567184ee79e402e8bd31973cd in term 3.
I20260926 14:50:32.177296 12038 leader_election.cc:302] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1deca22567184ee79e402e8bd31973cd, cdf14746a0d94d4488b627b215cf1100; no voters: 31006d7f535a4a9f8ac7b9faf17be594
I20260926 14:50:32.177418 12592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:50:32.177548 12592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [term 3 LEADER]: Becoming Leader. State: Replica: 1deca22567184ee79e402e8bd31973cd, State: Running, Role: LEADER
I20260926 14:50:32.177644 12592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } }
I20260926 14:50:32.178025 12595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1deca22567184ee79e402e8bd31973cd. Latest consensus state: current_term: 3 leader_uuid: "1deca22567184ee79e402e8bd31973cd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:32.178126 12595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:32.178292 12597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:32.178870 12597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:32.178954 12597 catalog_manager.cc:1259] Loaded cluster ID: 9b92355240654b549ff0a0d9722fbcca
I20260926 14:50:32.178990 12597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:32.179183 12597 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:32.179268 12597 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Loaded TSK: 0
I20260926 14:50:32.179431 12597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:32.229454 12054 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:32.231145 12054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:32.232116 11878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [term 3 FOLLOWER]: Refusing update from remote peer 1deca22567184ee79e402e8bd31973cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 14:50:32.232177 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.232058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44644)
0926 14:50:32.232111 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.232145 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:50:32.232369 12595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 14:50:32.232530 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.232418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44658)
0926 14:50:32.232462 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.232506 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:50:32.232612 12040 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd -> Peer 31006d7f535a4a9f8ac7b9faf17be594 (127.5.15.190:38521): Couldn't send request to peer 31006d7f535a4a9f8ac7b9faf17be594. Status: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:50:32.232981 12599 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1deca22567184ee79e402e8bd31973cd. Latest consensus state: current_term: 3 leader_uuid: "1deca22567184ee79e402e8bd31973cd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:32.233101 12599 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:32.233352 12598 mvcc.cc:205] Tried to move back new op lower bound from 7333618615220961280 to 7333618615000285184. Current Snapshot: MvccSnapshot[applied={T|T < 7333618615220961280}]
I20260926 14:50:32.233476 12600 mvcc.cc:205] Tried to move back new op lower bound from 7333618615220961280 to 7333618615000285184. Current Snapshot: MvccSnapshot[applied={T|T < 7333618615220961280}]
I20260926 14:50:32.233435 12592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1deca22567184ee79e402e8bd31973cd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:32.233521 12592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:32.233645 12599 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1deca22567184ee79e402e8bd31973cd" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "31006d7f535a4a9f8ac7b9faf17be594" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38521 } } peers { permanent_uuid: "cdf14746a0d94d4488b627b215cf1100" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 43051 } attrs { promote: false } } peers { permanent_uuid: "1deca22567184ee79e402e8bd31973cd" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 41809 } attrs { promote: false } } }
I20260926 14:50:32.233742 12599 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf14746a0d94d4488b627b215cf1100 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:50:32.238080 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.235255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46617 (local address 127.0.0.1:47078)
0926 14:50:32.235359 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.235369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.235413 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.235740 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.235741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.235743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:32.235815 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.235816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.236446 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.236447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.237188 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.237189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.237817 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.237818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.237888 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.237893 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.237921 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.237957 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":20,"thread_start_us":58,"threads_started":1}
I20260926 14:50:32.238338 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.235343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.5.15.129:46617)
0926 14:50:32.235604 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.235606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.235621 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:50:32.235642 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.235643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.235651 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.235735 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:32.235892 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.236365 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.237278 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.237725 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.238199 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.238214 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.238233 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.238271 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":59,"threads_started":1}
I20260926 14:50:32.238519 12169 tablet_service.cc:1467] Processing CreateTablet for tablet dad45c96ddfd43609159820f7d92a240 (DEFAULT_TABLE table=test-table [id=b407451793f449a39876783ebe8862f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:32.238654 12169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad45c96ddfd43609159820f7d92a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:32.239573 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.235236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37389 (local address 127.0.0.1:56742)
0926 14:50:32.235292 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.235303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.235360 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.236711 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.236712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.236713 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:32.236743 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.236744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.237620 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.237621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.238730 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.238732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.239320 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.239321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.239378 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.239382 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.239459 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.239511 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:50:32.239586 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.235482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.5.15.131:37389)
0926 14:50:32.236568 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.236570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.236576 (+ 6us) server_negotiation.cc:407] Connection header received
0926 14:50:32.236599 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.236600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.236610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.236798 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:32.236831 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.237542 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.238822 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.239212 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.239317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.239327 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.239466 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.239522 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":49,"threads_started":1}
I20260926 14:50:32.239928 12610 tablet_bootstrap.cc:492] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0: Bootstrap starting.
I20260926 14:50:32.239992 12441 tablet_service.cc:1467] Processing CreateTablet for tablet dad45c96ddfd43609159820f7d92a240 (DEFAULT_TABLE table=test-table [id=b407451793f449a39876783ebe8862f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:32.240118 12441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad45c96ddfd43609159820f7d92a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:32.240290 12610 tablet_bootstrap.cc:654] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:32.240906 12610 tablet_bootstrap.cc:492] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0: No bootstrap required, opened a new log
I20260926 14:50:32.240968 12610 ts_tablet_manager.cc:1397] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:32.241122 12610 raft_consensus.cc:348] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.241195 12610 raft_consensus.cc:374] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:32.241227 12610 raft_consensus.cc:729] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787cbee12fb34ed49903870502dcd5e0, State: Initialized, Role: FOLLOWER
I20260926 14:50:32.241277 12610 consensus_queue.cc:260] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.241461 12610 ts_tablet_manager.cc:1428] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:32.241665 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.241576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:56191)
0926 14:50:32.241617 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.241640 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 14:50:32.241926 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.241839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:47203)
0926 14:50:32.241877 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.241898 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 14:50:32.242007 12235 heartbeater.cc:503] Master 127.5.15.188:41809 was elected leader, sending a full tablet report...
I20260926 14:50:32.242035 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.235325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35939 (local address 127.0.0.1:46264)
0926 14:50:32.235484 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.235492 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.235528 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.235736 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.235737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.235739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:32.235812 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.235815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.236517 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.236518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.237752 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.237754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.241629 (+ 3875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.241811 (+ 182us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.241877 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.241883 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.241934 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.241976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260926 14:50:32.242151 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.235493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.5.15.130:35939)
0926 14:50:32.235594 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.235596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.235603 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:32.235626 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.235627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.235644 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.235915 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:32.235956 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.236429 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.237869 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.240334 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.240394 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.240422 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.241997 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.242036 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
I20260926 14:50:32.242195 12613 tablet_bootstrap.cc:492] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Bootstrap starting.
I20260926 14:50:32.242326 12305 tablet_service.cc:1467] Processing CreateTablet for tablet dad45c96ddfd43609159820f7d92a240 (DEFAULT_TABLE table=test-table [id=b407451793f449a39876783ebe8862f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:32.242434 12305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad45c96ddfd43609159820f7d92a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:32.242547 12613 tablet_bootstrap.cc:654] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:32.243182 12613 tablet_bootstrap.cc:492] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: No bootstrap required, opened a new log
I20260926 14:50:32.243239 12613 ts_tablet_manager.cc:1397] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:32.243336 12613 raft_consensus.cc:348] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.243403 12613 raft_consensus.cc:374] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:32.243423 12613 raft_consensus.cc:729] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce4cabce51e448fbc67b1f8e7a0cc45, State: Initialized, Role: FOLLOWER
I20260926 14:50:32.243470 12613 consensus_queue.cc:260] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.243628 12613 ts_tablet_manager.cc:1428] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:32.243970 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.243853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:52869)
0926 14:50:32.243920 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.243945 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 14:50:32.244045 12507 heartbeater.cc:503] Master 127.5.15.188:41809 was elected leader, sending a full tablet report...
I20260926 14:50:32.244248 12616 tablet_bootstrap.cc:492] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: Bootstrap starting.
I20260926 14:50:32.244601 12616 tablet_bootstrap.cc:654] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:32.245129 12616 tablet_bootstrap.cc:492] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: No bootstrap required, opened a new log
I20260926 14:50:32.245165 12616 ts_tablet_manager.cc:1397] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:32.245323 12616 raft_consensus.cc:348] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.245373 12616 raft_consensus.cc:374] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:32.245416 12616 raft_consensus.cc:729] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ee595e906f467a990314e743cecc93, State: Initialized, Role: FOLLOWER
I20260926 14:50:32.245481 12616 consensus_queue.cc:260] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.245685 12616 ts_tablet_manager.cc:1428] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:32.245924 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.245803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:44103)
0926 14:50:32.245868 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.245897 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 14:50:32.246121 12371 heartbeater.cc:503] Master 127.5.15.188:41809 was elected leader, sending a full tablet report...
I20260926 14:50:32.246372 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.246168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:60371)
0926 14:50:32.246296 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.246347 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 14:50:32.508118 12612 raft_consensus.cc:482] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:32.508260 12612 raft_consensus.cc:504] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.508646 12612 leader_election.cc:288] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389), 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:32.512739 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.509046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:39523 (local address 127.5.15.130:35939)
0926 14:50:32.509098 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.509101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.509214 (+ 113us) server_negotiation.cc:407] Connection header received
0926 14:50:32.509274 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.509275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.509287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.509339 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:32.509495 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.510214 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.511823 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.512374 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.512452 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.512461 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.512632 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.512670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:32.512739 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.509037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:43271 (local address 127.5.15.131:37389)
0926 14:50:32.509111 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.509113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.509121 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:32.509154 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.509155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.509166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.509233 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:32.509448 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.510221 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.511823 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.512374 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.512452 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.512461 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.512632 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.512670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:50:32.512794 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.508826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:37389 (local address 127.5.15.129:43271)
0926 14:50:32.508883 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.508895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.508959 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.509261 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.509262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.509265 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:32.509374 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.509377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.510301 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.510302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.511736 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.511739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.512465 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.512467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.512548 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.512555 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.512610 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.512675 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":73}
I20260926 14:50:32.512799 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.508955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35939 (local address 127.5.15.129:39523)
0926 14:50:32.509138 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.509145 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.509202 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.509365 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:32.509366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:32.509379 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:32.509425 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.509426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.510300 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.510301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.511735 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:32.511739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.512485 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.512486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:32.512550 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.512553 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:32.512610 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:50:32.512674 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":133,"thread_start_us":114,"threads_started":1}
I20260926 14:50:32.513105 12325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad45c96ddfd43609159820f7d92a240" candidate_uuid: "787cbee12fb34ed49903870502dcd5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ee595e906f467a990314e743cecc93" is_pre_election: true
I20260926 14:50:32.513109 12461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad45c96ddfd43609159820f7d92a240" candidate_uuid: "787cbee12fb34ed49903870502dcd5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" is_pre_election: true
I20260926 14:50:32.513206 12325 raft_consensus.cc:2454] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 787cbee12fb34ed49903870502dcd5e0 in term 0.
I20260926 14:50:32.513233 12461 raft_consensus.cc:2454] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 787cbee12fb34ed49903870502dcd5e0 in term 0.
I20260926 14:50:32.513374 12125 leader_election.cc:302] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ee595e906f467a990314e743cecc93, 787cbee12fb34ed49903870502dcd5e0; no voters:
I20260926 14:50:32.513507 12612 raft_consensus.cc:2781] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:32.513549 12612 raft_consensus.cc:482] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:32.513578 12612 raft_consensus.cc:3037] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:32.514251 12612 raft_consensus.cc:504] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.514374 12612 leader_election.cc:288] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [CANDIDATE]: Term 1 election: Requested vote from peers 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389), 37ee595e906f467a990314e743cecc93 (127.5.15.130:35939)
I20260926 14:50:32.514509 12461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad45c96ddfd43609159820f7d92a240" candidate_uuid: "787cbee12fb34ed49903870502dcd5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45"
I20260926 14:50:32.514526 12325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad45c96ddfd43609159820f7d92a240" candidate_uuid: "787cbee12fb34ed49903870502dcd5e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37ee595e906f467a990314e743cecc93"
I20260926 14:50:32.514583 12461 raft_consensus.cc:3037] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:32.514592 12325 raft_consensus.cc:3037] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:32.515298 12325 raft_consensus.cc:2454] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 787cbee12fb34ed49903870502dcd5e0 in term 1.
I20260926 14:50:32.515411 12461 raft_consensus.cc:2454] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 787cbee12fb34ed49903870502dcd5e0 in term 1.
I20260926 14:50:32.515465 12125 leader_election.cc:302] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ee595e906f467a990314e743cecc93, 787cbee12fb34ed49903870502dcd5e0; no voters:
I20260926 14:50:32.515551 12612 raft_consensus.cc:2781] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:32.515722 12612 raft_consensus.cc:686] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 787cbee12fb34ed49903870502dcd5e0, State: Running, Role: LEADER
I20260926 14:50:32.515901 12612 consensus_queue.cc:237] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } }
I20260926 14:50:32.516109 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.515946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:35861)
0926 14:50:32.516062 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.516088 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 14:50:32.516607 12054 catalog_manager.cc:5495] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "787cbee12fb34ed49903870502dcd5e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "787cbee12fb34ed49903870502dcd5e0" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:32.589859 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.587746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.5.15.129:46617)
0926 14:50:32.587810 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:32.587812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:32.587820 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:32.587859 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:32.587859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:32.587880 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:32.587963 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:32.588085 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.588556 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.589234 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:32.589414 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:32.589464 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:32.589590 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:32.589687 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:32.589738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:32.589780 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:50:32.589798 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:32.590924 12325 raft_consensus.cc:1261] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Refusing update from remote peer 787cbee12fb34ed49903870502dcd5e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:32.591037 12461 raft_consensus.cc:1261] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Refusing update from remote peer 787cbee12fb34ed49903870502dcd5e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:32.591192 12612 consensus_queue.cc:1035] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37ee595e906f467a990314e743cecc93" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:32.591357 12620 consensus_queue.cc:1035] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce4cabce51e448fbc67b1f8e7a0cc45" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:32.592082 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.591958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:57421)
0926 14:50:32.592005 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.592036 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:50:32.592124 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.592024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:55563)
0926 14:50:32.592064 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.592090 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 14:50:32.593168 12624 mvcc.cc:205] Tried to move back new op lower bound from 7333618616690270208 to 7333618616385650688. Current Snapshot: MvccSnapshot[applied={T|T < 7333618616690270208}]
I20260926 14:50:32.593492 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.593389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.129:54933)
0926 14:50:32.593431 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.593460 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 14:50:32.593509 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.593382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.131:34067)
0926 14:50:32.593450 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.593482 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:50:32.593597 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.593509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.5.15.130:57821)
0926 14:50:32.593549 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.593571 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 14:50:32.594092 12626 mvcc.cc:205] Tried to move back new op lower bound from 7333618616690270208 to 7333618616385650688. Current Snapshot: MvccSnapshot[applied={T|T < 7333618616690270208}]
I20260926 14:50:32.595928 12054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:32.596024 12054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:32.597257 12054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1deca22567184ee79e402e8bd31973cd: Sending DeleteTablet for 3 replicas of tablet dad45c96ddfd43609159820f7d92a240
I20260926 14:50:32.597647 12441 tablet_service.cc:1514] Processing DeleteTablet for tablet dad45c96ddfd43609159820f7d92a240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:32 UTC) from {username='oryx', principal='oryx/127.5.15.188@KRBTEST.COM'} at 127.0.0.1:56742
I20260926 14:50:32.597645 12169 tablet_service.cc:1514] Processing DeleteTablet for tablet dad45c96ddfd43609159820f7d92a240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:32 UTC) from {username='oryx', principal='oryx/127.5.15.188@KRBTEST.COM'} at 127.0.0.1:47078
I20260926 14:50:32.597826 12305 tablet_service.cc:1514] Processing DeleteTablet for tablet dad45c96ddfd43609159820f7d92a240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:32 UTC) from {username='oryx', principal='oryx/127.5.15.188@KRBTEST.COM'} at 127.0.0.1:46264
I20260926 14:50:32.597832 12633 tablet_replica.cc:330] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0: stopping tablet replica
I20260926 14:50:32.597885 12633 raft_consensus.cc:2229] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:32.597980 12633 raft_consensus.cc:2258] T dad45c96ddfd43609159820f7d92a240 P 787cbee12fb34ed49903870502dcd5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:32.598061 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 12106
I20260926 14:50:32.598091 12634 tablet_replica.cc:330] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: stopping tablet replica
I20260926 14:50:32.598135 12634 raft_consensus.cc:2229] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:32.598173 12634 raft_consensus.cc:2258] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:32.598853 12634 ts_tablet_manager.cc:1905] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:32.600317 12634 ts_tablet_manager.cc:1918] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:32.600354 12634 log.cc:1196] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/dad45c96ddfd43609159820f7d92a240
I20260926 14:50:32.600598 12634 ts_tablet_manager.cc:1939] T dad45c96ddfd43609159820f7d92a240 P 5ce4cabce51e448fbc67b1f8e7a0cc45: Deleting consensus metadata
I20260926 14:50:32.600951 12041 catalog_manager.cc:4841] TS 5ce4cabce51e448fbc67b1f8e7a0cc45 (127.5.15.131:37389): tablet dad45c96ddfd43609159820f7d92a240 (table test-table [id=b407451793f449a39876783ebe8862f5]) successfully deleted
I20260926 14:50:32.602843 12635 tablet_replica.cc:330] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: stopping tablet replica
I20260926 14:50:32.602914 12635 raft_consensus.cc:2229] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:32.602964 12635 raft_consensus.cc:2258] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:32.603224 12635 ts_tablet_manager.cc:1905] T dad45c96ddfd43609159820f7d92a240 P 37ee595e906f467a990314e743cecc93: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:50:32.604182 12606 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:32.603977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46617 (local address 127.0.0.1:47102)
0926 14:50:32.604037 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.604048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:32.604083 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:32.604128 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:46617: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 14:50:32.604303 12040 catalog_manager.cc:4584] TS 787cbee12fb34ed49903870502dcd5e0 (127.5.15.129:46617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dad45c96ddfd43609159820f7d92a240: Network error: Client connection negotiation failed: client connection to 127.5.15.129:46617: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260926 14:50:32.604460 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 12242
I20260926 14:50:32.609977 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.609875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35939 (local address 127.0.0.1:46270)
0926 14:50:32.609925 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.609952 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.130:35939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:50:32.610097 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 12378
I20260926 14:50:32.614944 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.610230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44670)
0926 14:50:32.614887 (+ 4657us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.614918 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4640}
I20260926 14:50:32.615268 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:32.615175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38521 (local address 127.0.0.1:44678)
0926 14:50:32.615221 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:32.615245 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:50:32.615763 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11734
I20260926 14:50:32.620726 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 11922
2026-09-26T14:50:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6535 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:32 dist-test-slave-s4p1 krb5kdc[12638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:32 dist-test-slave-s4p1 krb5kdc[12638](info): set up 2 sockets
Sep 26 14:50:32 dist-test-slave-s4p1 krb5kdc[12638](info): commencing operation
krb5kdc: starting...
W20260926 14:50:33.639204 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:33 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab.
Entry for principal oryx/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab.
I20260926 14:50:33.747251 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38433
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab
--principal=oryx/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:33.822546 12651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:33.822734 12651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:33.822818 12651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:33.823953 12651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:33.823989 12651 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:50:33.824002 12651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:33.824049 12651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:33.824072 12651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:33.824095 12651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:33.825493 12651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.190.keytab
--principal=oryx/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:33.825726 12651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:33.825986 12651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:33.830807 12651 server_base.cc:1034] running on GCE node
W20260926 14:50:33.830830 12657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.830849 12656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:33.831023 12659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:33.831269 12651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:33.831554 12651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:33.832701 12651 hybrid_clock.cc:648] HybridClock initialized: now 1790434233832696 us; error 34 us; skew 500 ppm
Sep 26 14:50:33 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:33.834990 12651 init.cc:377] Logged in from keytab as oryx/127.5.15.190@KRBTEST.COM (short username oryx)
I20260926 14:50:33.835398 12651 webserver.cc:460] Webserver started at http://127.5.15.190:40227/ using document root <none> and password file <none>
I20260926 14:50:33.835599 12651 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:33.835664 12651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:33.835799 12651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:33.836781 12651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "d8c9292afadb426ba575c1210f483195"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-s4p1"
server_key: "6d9bbcffd1d56b651a2fbeb020a8377f"
server_key_iv: "76a10c8749f32295984e1a277636da69"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.837121 12651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "d8c9292afadb426ba575c1210f483195"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-s4p1"
server_key: "6d9bbcffd1d56b651a2fbeb020a8377f"
server_key_iv: "76a10c8749f32295984e1a277636da69"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.838387 12651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:33.839123 12666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.839269 12651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:33.839359 12651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "d8c9292afadb426ba575c1210f483195"
format_stamp: "Formatted at 2026-09-26 14:50:33 on dist-test-slave-s4p1"
server_key: "6d9bbcffd1d56b651a2fbeb020a8377f"
server_key_iv: "76a10c8749f32295984e1a277636da69"
server_key_version: "encryptionkey@0"
I20260926 14:50:33.839423 12651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:33.854210 12651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:33.856034 12651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:33.856201 12651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:33.860597 12651 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:38433
I20260926 14:50:33.860675 12728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:38433 every 8 connection(s)
I20260926 14:50:33.860997 12651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:33.861176 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 12651
I20260926 14:50:33.861291 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:33.861515 5182 external_mini_cluster.cc:1428] Setting key 47b196d5fbff414f3005949a0a821d55
I20260926 14:50:33.861774 12729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:33.864859 12729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Bootstrap starting.
I20260926 14:50:33.865522 12729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:33.865795 12729 log.cc:824] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:33.866485 12729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: No bootstrap required, opened a new log
Sep 26 14:50:33 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:33.868333 12729 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c9292afadb426ba575c1210f483195" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38433 } }
I20260926 14:50:33.868533 12729 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:33.868572 12729 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c9292afadb426ba575c1210f483195, State: Initialized, Role: FOLLOWER
I20260926 14:50:33.868709 12729 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c9292afadb426ba575c1210f483195" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38433 } }
I20260926 14:50:33.868835 12729 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:33.868877 12729 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:33.868947 12729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:33.869657 12729 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c9292afadb426ba575c1210f483195" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38433 } }
I20260926 14:50:33.869762 12729 leader_election.cc:302] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c9292afadb426ba575c1210f483195; no voters:
I20260926 14:50:33.869917 12729 leader_election.cc:288] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:33.869962 12734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:33.870137 12734 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [term 1 LEADER]: Becoming Leader. State: Replica: d8c9292afadb426ba575c1210f483195, State: Running, Role: LEADER
I20260926 14:50:33.870149 12729 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:33.870237 12734 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c9292afadb426ba575c1210f483195" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38433 } }
I20260926 14:50:33.870602 12734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c9292afadb426ba575c1210f483195. Latest consensus state: current_term: 1 leader_uuid: "d8c9292afadb426ba575c1210f483195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c9292afadb426ba575c1210f483195" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38433 } } }
I20260926 14:50:33.870610 12735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c9292afadb426ba575c1210f483195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c9292afadb426ba575c1210f483195" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38433 } } }
I20260926 14:50:33.870822 12735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:33.871012 12734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:33.871204 12743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:33.871771 12743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:33.872349 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:33.862038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.5.15.190:38433)
0926 14:50:33.862209 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:33.862217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:33.862247 (+ 30us) server_negotiation.cc:407] Connection header received
0926 14:50:33.862865 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:33.862872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:33.862918 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:33.863026 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:33.863777 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.864607 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.865627 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:33.866068 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:33.866149 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:33.868452 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:33.868455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:33.868455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:33.868466 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:33.870440 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.870949 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.870952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.870952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.871065 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:33.871289 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:33.871290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:33.871291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:33.871660 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:33.871827 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:33.871948 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 14:50:33.872075 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260926 14:50:33.875205 12743 catalog_manager.cc:1347] Generated new cluster ID: f66f315fa8db45ee8c1284061d73459a
I20260926 14:50:33.875274 12743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:33.884617 12743 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:33.885306 12743 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:33.898720 12743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Generated new TSK 0
I20260926 14:50:33.898931 12743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab.
Entry for principal oryx/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab.
I20260926 14:50:33.925338 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38433
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:34.001305 12755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:34.001484 12755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:34.001506 12755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:34.002595 12755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:34.002631 12755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:34.002650 12755 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:34.002701 12755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:34.004055 12755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:38433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:34.004222 12755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:34.004436 12755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:34.009035 12761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.009080 12755 server_base.cc:1034] running on GCE node
W20260926 14:50:34.009053 12760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.009186 12763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.009482 12755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:34.009682 12755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:34.010807 12755 hybrid_clock.cc:648] HybridClock initialized: now 1790434234010792 us; error 25 us; skew 500 ppm
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:34.013206 12755 init.cc:377] Logged in from keytab as oryx/127.5.15.129@KRBTEST.COM (short username oryx)
I20260926 14:50:34.013609 12755 webserver.cc:460] Webserver started at http://127.5.15.129:42981/ using document root <none> and password file <none>
I20260926 14:50:34.013835 12755 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:34.013901 12755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:34.014055 12755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:34.015110 12755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "765383c591cb4d27bf165171f598d3f9"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "da5109f11d76c383225c8d60127f5cdb"
server_key_iv: "42512e29c36eaa312076bae613c32d55"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.015456 12755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "765383c591cb4d27bf165171f598d3f9"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "da5109f11d76c383225c8d60127f5cdb"
server_key_iv: "42512e29c36eaa312076bae613c32d55"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.016716 12755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:34.017396 12770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.017563 12755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:34.017652 12755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "765383c591cb4d27bf165171f598d3f9"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "da5109f11d76c383225c8d60127f5cdb"
server_key_iv: "42512e29c36eaa312076bae613c32d55"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.017731 12755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:34.039649 12755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:34.041363 12755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:34.041515 12755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:34.042092 12755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:34.042169 12755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.042210 12755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:34.042235 12755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:34.048902 12755 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:40547
I20260926 14:50:34.049258 12755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:34.049315 12885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:40547 every 8 connection(s)
I20260926 14:50:34.049623 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 12755
I20260926 14:50:34.049718 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:34.049934 5182 external_mini_cluster.cc:1428] Setting key f07b23db375ce9a90876a74a385576f1
I20260926 14:50:34.052371 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.042530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:37217 (local address 127.5.15.190:38433)
0926 14:50:34.042569 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.042571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.042827 (+ 256us) server_negotiation.cc:407] Connection header received
0926 14:50:34.043359 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.043361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.043372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.043437 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.044296 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.044800 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.046341 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.046677 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.046770 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.049287 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.049289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.049289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.049296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.050991 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.051383 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.051384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.051384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.051441 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.051702 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.051703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.051703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.051817 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.051889 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.052188 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.052279 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:50:34.052578 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.042527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.5.15.129:37217)
0926 14:50:34.042686 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.042719 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.043255 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.043501 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.043504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.043917 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.044191 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.044202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.044896 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.044897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.046249 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.046253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.046804 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.046805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.046984 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.047451 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.047461 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.049193 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.051080 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.051087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.051089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.051306 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.051515 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.051516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.051516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.051626 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.051881 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.051887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.052025 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.052165 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.052317 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260926 14:50:34.053797 12886 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38433
I20260926 14:50:34.053913 12886 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.054164 12886 heartbeater.cc:511] Master 127.5.15.190:38433 requested a full tablet report, sending...
I20260926 14:50:34.054697 12682 ts_manager.cc:194] Registered new tserver with Master: 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547)
I20260926 14:50:34.055552 12682 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:37217
WARNING: no policy specified for oryx/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.130@KRBTEST.COM" created.
I20260926 14:50:34.060245 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.053950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:34901 (local address 127.5.15.190:38433)
0926 14:50:34.053989 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.053991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.054086 (+ 95us) server_negotiation.cc:407] Connection header received
0926 14:50:34.054177 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.054178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.054188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.054217 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.054594 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.055478 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.056422 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.056618 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.056701 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.057605 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.057606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.057607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.057613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.059158 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.059499 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.059500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.059500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.059550 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.059750 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.059751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.059751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.059854 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.059926 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.060079 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.060144 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":22}
I20260926 14:50:34.060266 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.053899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.5.15.129:34901)
0926 14:50:34.054016 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.054027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.054111 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.054273 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.054273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.054422 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.054499 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.054501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.055591 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.055592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.056315 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.056318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.056703 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.056704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.056776 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.057239 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.057243 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.057493 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.059253 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.059254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.059254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.059416 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.059624 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.059625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.059626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.059680 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.059919 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.059920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.059961 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.060048 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.060142 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab.
Entry for principal oryx/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab.
I20260926 14:50:34.073559 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38433
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:34.152992 12895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:34.153168 12895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:34.153189 12895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:34.154384 12895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:34.154423 12895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:34.154443 12895 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:34.154492 12895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:34.155934 12895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:38433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:34.156169 12895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:34.156383 12895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:34.161588 12903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.161588 12900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.161705 12901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.161746 12895 server_base.cc:1034] running on GCE node
I20260926 14:50:34.162024 12895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:34.162252 12895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:34.163410 12895 hybrid_clock.cc:648] HybridClock initialized: now 1790434234163391 us; error 35 us; skew 500 ppm
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:34.165802 12895 init.cc:377] Logged in from keytab as oryx/127.5.15.130@KRBTEST.COM (short username oryx)
I20260926 14:50:34.166254 12895 webserver.cc:460] Webserver started at http://127.5.15.130:35653/ using document root <none> and password file <none>
I20260926 14:50:34.166433 12895 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:34.166471 12895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:34.166561 12895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:34.167584 12895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "223ad00f8e774a7b81c42444192a3601"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "b246f2f35262ceb8a6d4dfbeb966f609"
server_key_iv: "7a441ebb7af4eba710870e6aaf3e642e"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.167869 12895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "223ad00f8e774a7b81c42444192a3601"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "b246f2f35262ceb8a6d4dfbeb966f609"
server_key_iv: "7a441ebb7af4eba710870e6aaf3e642e"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.169026 12895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:50:34.169658 12910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.169838 12895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:34.169938 12895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "223ad00f8e774a7b81c42444192a3601"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "b246f2f35262ceb8a6d4dfbeb966f609"
server_key_iv: "7a441ebb7af4eba710870e6aaf3e642e"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.170017 12895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:34.185684 12895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:34.187923 12895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:34.188095 12895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:34.188606 12895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:34.188664 12895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.188692 12895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:34.188721 12895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:34.195422 12895 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:41479
I20260926 14:50:34.195771 13025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:41479 every 8 connection(s)
I20260926 14:50:34.195786 12895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:34.197672 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 12895
I20260926 14:50:34.197793 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:34.197980 5182 external_mini_cluster.cc:1428] Setting key 986cd8d97848e4928cfef594934cdc23
I20260926 14:50:34.198174 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.188961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:50627 (local address 127.5.15.190:38433)
0926 14:50:34.189014 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.189016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.189192 (+ 176us) server_negotiation.cc:407] Connection header received
0926 14:50:34.189659 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.189661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.189674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.189747 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.190660 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.191212 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.192149 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.192342 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.192420 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.195196 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.195198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.195198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.195205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.196798 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.197185 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.197187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.197187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.197242 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.197502 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.197503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.197503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.197634 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.197704 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.198009 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.198084 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:50:34.198352 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.188911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.5.15.130:50627)
0926 14:50:34.189085 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.189115 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.189560 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.189863 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.189867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.190271 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.190551 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.190557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.191322 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.191323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.192049 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.192051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.192428 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.192429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.192572 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.193050 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.193057 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.195103 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.196886 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.196891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.196893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.197107 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.197320 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.197320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.197321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.197427 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.197708 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.197714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.197859 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.197987 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.198135 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260926 14:50:34.198828 13026 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38433
I20260926 14:50:34.198907 13026 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.199121 13026 heartbeater.cc:511] Master 127.5.15.190:38433 requested a full tablet report, sending...
I20260926 14:50:34.199535 12682 ts_manager.cc:194] Registered new tserver with Master: 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479)
I20260926 14:50:34.200011 12682 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:50627
I20260926 14:50:34.205194 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.199761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:36613 (local address 127.5.15.190:38433)
0926 14:50:34.199797 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.199799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.199870 (+ 71us) server_negotiation.cc:407] Connection header received
0926 14:50:34.199943 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.199944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.199954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.200000 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.200405 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.200989 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.201692 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.201892 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.201996 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.202865 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.202866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.202866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.202873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.204137 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.204435 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.204436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.204437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.204484 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.204670 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.204671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.204672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.204803 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.204890 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.205042 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.205113 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:34.205207 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.199715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.5.15.130:36613)
0926 14:50:34.199824 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.199833 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.199883 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.200042 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.200042 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.200235 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.200308 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.200310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.201083 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.201083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.201606 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.201608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.201998 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.202000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.202060 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.202475 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.202479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.202711 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.204215 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.204216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.204216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.204368 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.204553 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.204554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.204554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.204604 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.204894 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.204895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.204937 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.205019 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.205112 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab.
Entry for principal oryx/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab.
I20260926 14:50:34.223021 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38433
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:34.299774 13035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:34.299966 13035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:34.299988 13035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:34.301061 13035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:34.301110 13035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:34.301131 13035 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:50:34.301147 13035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:34.302486 13035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.15.190:38433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:34.302649 13035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:34.302814 13035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:34.307864 13040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.307853 13043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:34.307904 13041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:34.308061 13035 server_base.cc:1034] running on GCE node
I20260926 14:50:34.308254 13035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:34.308422 13035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:34.309558 13035 hybrid_clock.cc:648] HybridClock initialized: now 1790434234309540 us; error 33 us; skew 500 ppm
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:34.312091 13035 init.cc:377] Logged in from keytab as oryx/127.5.15.131@KRBTEST.COM (short username oryx)
I20260926 14:50:34.312495 13035 webserver.cc:460] Webserver started at http://127.5.15.131:42689/ using document root <none> and password file <none>
I20260926 14:50:34.312680 13035 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:34.312716 13035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:34.312796 13035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:34.313671 13035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "f8938691ab2a4b0bbf1358d54c351c39"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "87bce91d38db8014ce1eecb9e2af26c7"
server_key_iv: "f96745ceca8f2b4e095c9e1eb3324eda"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.313946 13035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "f8938691ab2a4b0bbf1358d54c351c39"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "87bce91d38db8014ce1eecb9e2af26c7"
server_key_iv: "f96745ceca8f2b4e095c9e1eb3324eda"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.315124 13035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:34.315721 13050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.315907 13035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:34.315968 13035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "f8938691ab2a4b0bbf1358d54c351c39"
format_stamp: "Formatted at 2026-09-26 14:50:34 on dist-test-slave-s4p1"
server_key: "87bce91d38db8014ce1eecb9e2af26c7"
server_key_iv: "f96745ceca8f2b4e095c9e1eb3324eda"
server_key_version: "encryptionkey@0"
I20260926 14:50:34.316051 13035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:34.323863 13035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:34.326300 13035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:34.326452 13035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:34.327008 13035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:34.327069 13035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.327108 13035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:34.327140 13035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.333441 13035 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:42397
I20260926 14:50:34.333884 13035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:34.334187 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13035
I20260926 14:50:34.334296 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:34.334482 5182 external_mini_cluster.cc:1428] Setting key ad96c33712f1aa3ee434c693c8850ced
I20260926 14:50:34.335842 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:42397 every 8 connection(s)
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:34.343886 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.334685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:46059 (local address 127.5.15.190:38433)
0926 14:50:34.334732 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.334734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.334995 (+ 261us) server_negotiation.cc:407] Connection header received
0926 14:50:34.335634 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.335635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.335658 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.335703 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.336443 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.337131 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.338637 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.338988 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.339064 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.341149 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.341150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.341151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.341157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.342579 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.342985 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.342986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.342987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.343044 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.343268 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.343269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.343269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.343374 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.343442 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.343706 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.343797 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 14:50:34.344066 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.334114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.5.15.131:46059)
0926 14:50:34.334888 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.334928 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.335521 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.335810 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.335813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.336142 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.336361 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.336367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.337236 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.337237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.338535 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.338538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.339073 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.339074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.339222 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.339716 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.339723 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.341056 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.342652 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.342658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.342660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.342916 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.343104 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.343105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.343105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.343207 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.343434 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.343439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.343578 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.343686 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.343847 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":74,"threads_started":1}
I20260926 14:50:34.344499 13165 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38433
I20260926 14:50:34.344601 13165 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:34.344761 13165 heartbeater.cc:511] Master 127.5.15.190:38433 requested a full tablet report, sending...
I20260926 14:50:34.345177 12682 ts_manager.cc:194] Registered new tserver with Master: f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:34.345659 12682 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:46059
I20260926 14:50:34.346297 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:34.351337 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.345433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.5.15.131:59073)
0926 14:50:34.345879 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.345890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.345962 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.346133 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.346134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.346344 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.346389 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.346391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347115 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.347616 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.347617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347981 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.348056 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.348471 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.348476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.348725 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.350369 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.350369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.350370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.350521 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.350737 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.350738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.350738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.350824 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.351071 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.351072 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.351102 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.351147 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.351242 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":74,"threads_started":1}
I20260926 14:50:34.351348 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.345785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:59073 (local address 127.5.15.190:38433)
0926 14:50:34.345838 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.345840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.345937 (+ 97us) server_negotiation.cc:407] Connection header received
0926 14:50:34.346024 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.346025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.346035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.346082 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.346470 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347027 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347701 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.347900 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.347969 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.348827 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.348828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.348828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.348835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.350299 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.350594 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.350596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.350596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.350662 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.350884 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.350885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.350886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.351006 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.351076 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.351174 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.351242 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:34.367610 12732 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:34.362229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.5.15.190:38433)
0926 14:50:34.362293 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.362295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.362304 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:34.362364 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.362365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.362377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.362442 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.363159 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.363635 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.364366 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.364545 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.364652 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.367540 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41706: BlockingRecv error: recv got EOF from 127.0.0.1:41706 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434234, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:34.374362 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.368361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.5.15.190:38433)
0926 14:50:34.368399 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.368401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.368488 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:50:34.368535 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.368536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.368543 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.368595 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.369003 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.369430 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.370110 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.370274 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.370322 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.372370 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.372371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.372371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.372376 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.373519 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.373834 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.373834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.373835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.373881 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.374002 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.374003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.374003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.374084 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.374147 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.374232 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.374299 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:34.375725 12682 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:34.377184 12682 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:34.383484 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.379298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:41479 (local address 127.0.0.1:46112)
0926 14:50:34.379627 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.379637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.379676 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.379928 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.379931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.379933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.379994 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.379996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.381367 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.381368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.382377 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.382379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.383278 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.383279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.383338 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.383342 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.383389 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.383418 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":7,"spinlock_wait_cycles":768,"thread_start_us":56,"threads_started":1}
I20260926 14:50:34.383530 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.379546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.5.15.129:40547)
0926 14:50:34.379696 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.379702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.379805 (+ 103us) server_negotiation.cc:407] Connection header received
0926 14:50:34.379843 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.379846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.379860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.379952 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.380180 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.380902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.382233 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.383066 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.383141 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.383218 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.383376 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.383416 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260926 14:50:34.383702 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.379473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.5.15.131:42397)
0926 14:50:34.380030 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.380034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.380043 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:34.380076 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.380077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.380091 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.380165 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.380359 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.381368 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.382487 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.383196 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.383257 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.383315 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.383568 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.383613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":80,"threads_started":1}
I20260926 14:50:34.384014 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.379474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.5.15.130:41479)
0926 14:50:34.379743 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.379746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.379755 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:34.379798 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.379800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.379818 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.379912 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.380163 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.381279 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.382490 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.383074 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.383179 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.383251 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.383906 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.383938 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":74,"threads_started":1}
I20260926 14:50:34.383486 13185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.379314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.0.0.1:53966)
0926 14:50:34.379742 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.379748 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.379783 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.380027 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.380028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.380030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.380082 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.380084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.380990 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.380991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.382146 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.382149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.383179 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.383180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.383260 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.383275 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.383349 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.383407 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":4,"thread_start_us":54,"threads_started":1}
I20260926 14:50:34.383860 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.379298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:42397 (local address 127.0.0.1:47934)
0926 14:50:34.379468 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.379505 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.379587 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.380230 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.380230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.380232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.380274 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.380275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.381432 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.381433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.382365 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.382368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.383369 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.383370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.383429 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.383433 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.383485 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.383525 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":1,"thread_start_us":55,"threads_started":1}
I20260926 14:50:34.384779 12960 tablet_service.cc:1467] Processing CreateTablet for tablet e9d44e39b5474577838589820d5a7bbd (DEFAULT_TABLE table=test-table [id=837d7d271828409cab2135dcbc97252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.385098 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9d44e39b5474577838589820d5a7bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.385006 13099 tablet_service.cc:1467] Processing CreateTablet for tablet e9d44e39b5474577838589820d5a7bbd (DEFAULT_TABLE table=test-table [id=837d7d271828409cab2135dcbc97252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.385344 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9d44e39b5474577838589820d5a7bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.388054 13191 tablet_bootstrap.cc:492] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Bootstrap starting.
I20260926 14:50:34.388368 12820 tablet_service.cc:1467] Processing CreateTablet for tablet e9d44e39b5474577838589820d5a7bbd (DEFAULT_TABLE table=test-table [id=837d7d271828409cab2135dcbc97252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.388648 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9d44e39b5474577838589820d5a7bbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.388669 13191 tablet_bootstrap.cc:654] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.388947 13191 log.cc:824] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:34.389243 13192 tablet_bootstrap.cc:492] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Bootstrap starting.
I20260926 14:50:34.389581 13191 tablet_bootstrap.cc:492] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: No bootstrap required, opened a new log
I20260926 14:50:34.389642 13191 ts_tablet_manager.cc:1397] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:34.391232 13191 raft_consensus.cc:348] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.391355 13191 raft_consensus.cc:374] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.391376 13191 raft_consensus.cc:729] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.391489 13191 consensus_queue.cc:260] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.391781 13191 ts_tablet_manager.cc:1428] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:34.391810 13026 heartbeater.cc:503] Master 127.5.15.190:38433 was elected leader, sending a full tablet report...
I20260926 14:50:34.391829 13195 tablet_bootstrap.cc:492] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Bootstrap starting.
I20260926 14:50:34.392405 13192 tablet_bootstrap.cc:654] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.392405 13195 tablet_bootstrap.cc:654] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.392724 13195 log.cc:824] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:34.392724 13192 log.cc:824] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:34.393368 13192 tablet_bootstrap.cc:492] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: No bootstrap required, opened a new log
I20260926 14:50:34.393431 13192 ts_tablet_manager.cc:1397] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 14:50:34.393540 13195 tablet_bootstrap.cc:492] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: No bootstrap required, opened a new log
I20260926 14:50:34.393596 13195 ts_tablet_manager.cc:1397] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:34.395454 13192 raft_consensus.cc:348] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.395656 13192 raft_consensus.cc:374] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.395690 13192 raft_consensus.cc:729] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8938691ab2a4b0bbf1358d54c351c39, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.395630 13195 raft_consensus.cc:348] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.395761 13195 raft_consensus.cc:374] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.395848 13195 raft_consensus.cc:729] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.395805 13192 consensus_queue.cc:260] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.395960 13195 consensus_queue.cc:260] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.396050 13192 ts_tablet_manager.cc:1428] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:34.396198 13195 ts_tablet_manager.cc:1428] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:34.396229 12886 heartbeater.cc:503] Master 127.5.15.190:38433 was elected leader, sending a full tablet report...
I20260926 14:50:34.396088 13165 heartbeater.cc:503] Master 127.5.15.190:38433 was elected leader, sending a full tablet report...
W20260926 14:50:34.446375 13027 tablet.cc:2369] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:34.550186 12887 tablet.cc:2369] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:50:34.586299 13166 tablet.cc:2369] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:50:34.609647 13198 raft_consensus.cc:482] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:34.609758 13198 raft_consensus.cc:504] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.610191 13198 leader_election.cc:288] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479), 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547)
I20260926 14:50:34.614032 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.610442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:47569 (local address 127.5.15.130:41479)
0926 14:50:34.610490 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.610492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.610626 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:50:34.610654 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.610656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.610667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.610716 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.610886 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.611646 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.613149 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.613682 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.613758 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.613766 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.613930 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.613966 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:50:34.614032 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.610439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:34245 (local address 127.5.15.129:40547)
0926 14:50:34.610484 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.610486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.610497 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:50:34.610556 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.610556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.610566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.610654 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.610877 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.611650 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.613144 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.613691 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.613758 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.613766 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.613930 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.613966 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:50:34.614058 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.610367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.5.15.131:34245)
0926 14:50:34.610426 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.610439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.610487 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.610661 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.610661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.610663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.610788 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.610791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.611728 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.611729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.613054 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.613056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.613766 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.613767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.613851 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.613855 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.613911 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.613950 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":73}
I20260926 14:50:34.614068 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.610374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:41479 (local address 127.5.15.131:47569)
0926 14:50:34.610540 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.610549 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.610602 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.610737 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.610737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.610738 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.610813 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.610814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.611728 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.611729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.613058 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.613059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.613766 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.613767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.613851 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.613855 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.613911 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.613950 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":93,"thread_start_us":93,"threads_started":1}
I20260926 14:50:34.614449 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d44e39b5474577838589820d5a7bbd" candidate_uuid: "f8938691ab2a4b0bbf1358d54c351c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765383c591cb4d27bf165171f598d3f9" is_pre_election: true
I20260926 14:50:34.614463 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d44e39b5474577838589820d5a7bbd" candidate_uuid: "f8938691ab2a4b0bbf1358d54c351c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223ad00f8e774a7b81c42444192a3601" is_pre_election: true
I20260926 14:50:34.614630 12979 raft_consensus.cc:2454] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8938691ab2a4b0bbf1358d54c351c39 in term 0.
I20260926 14:50:34.614626 12840 raft_consensus.cc:2454] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8938691ab2a4b0bbf1358d54c351c39 in term 0.
I20260926 14:50:34.614852 13054 leader_election.cc:302] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, f8938691ab2a4b0bbf1358d54c351c39; no voters:
I20260926 14:50:34.614969 13198 raft_consensus.cc:2781] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:34.615020 13198 raft_consensus.cc:482] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:34.615074 13198 raft_consensus.cc:3037] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.616001 13198 raft_consensus.cc:504] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.616147 13198 leader_election.cc:288] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [CANDIDATE]: Term 1 election: Requested vote from peers 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479), 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547)
I20260926 14:50:34.616313 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d44e39b5474577838589820d5a7bbd" candidate_uuid: "f8938691ab2a4b0bbf1358d54c351c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765383c591cb4d27bf165171f598d3f9"
I20260926 14:50:34.616307 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d44e39b5474577838589820d5a7bbd" candidate_uuid: "f8938691ab2a4b0bbf1358d54c351c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223ad00f8e774a7b81c42444192a3601"
I20260926 14:50:34.616384 12979 raft_consensus.cc:3037] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.616384 12840 raft_consensus.cc:3037] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.617190 12979 raft_consensus.cc:2454] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8938691ab2a4b0bbf1358d54c351c39 in term 1.
I20260926 14:50:34.617190 12840 raft_consensus.cc:2454] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8938691ab2a4b0bbf1358d54c351c39 in term 1.
I20260926 14:50:34.617352 13054 leader_election.cc:302] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, f8938691ab2a4b0bbf1358d54c351c39; no voters:
I20260926 14:50:34.617466 13198 raft_consensus.cc:2781] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:34.617609 13198 raft_consensus.cc:686] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 1 LEADER]: Becoming Leader. State: Replica: f8938691ab2a4b0bbf1358d54c351c39, State: Running, Role: LEADER
I20260926 14:50:34.617708 13198 consensus_queue.cc:237] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } }
I20260926 14:50:34.618489 12682 catalog_manager.cc:5495] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "f8938691ab2a4b0bbf1358d54c351c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:34.664505 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.662118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.5.15.131:42397)
0926 14:50:34.662195 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.662197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.662205 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:34.662244 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.662245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.662272 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.662349 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:34.662495 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.663024 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.663747 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.663934 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.664008 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.664172 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:34.664276 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:34.664358 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.664400 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.664419 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:34.666428 12840 raft_consensus.cc:1261] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Refusing update from remote peer f8938691ab2a4b0bbf1358d54c351c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:34.666428 12979 raft_consensus.cc:1261] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Refusing update from remote peer f8938691ab2a4b0bbf1358d54c351c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:34.666824 13201 consensus_queue.cc:1035] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:34.666965 13198 consensus_queue.cc:1035] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:34.669453 13204 mvcc.cc:205] Tried to move back new op lower bound from 7333618625190428672 to 7333618624994668544. Current Snapshot: MvccSnapshot[applied={T|T < 7333618625190428672}]
I20260926 14:50:34.669481 13205 mvcc.cc:205] Tried to move back new op lower bound from 7333618625190428672 to 7333618624994668544. Current Snapshot: MvccSnapshot[applied={T|T < 7333618625190428672}]
I20260926 14:50:34.671881 12683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:34.671957 12683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:34.672681 12683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Sending DeleteTablet for 3 replicas of tablet e9d44e39b5474577838589820d5a7bbd
I20260926 14:50:34.672959 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet e9d44e39b5474577838589820d5a7bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:34 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:46112
I20260926 14:50:34.672989 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet e9d44e39b5474577838589820d5a7bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:34 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:47934
I20260926 14:50:34.673013 12820 tablet_service.cc:1514] Processing DeleteTablet for tablet e9d44e39b5474577838589820d5a7bbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:34 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:53966
I20260926 14:50:34.673187 13215 tablet_replica.cc:330] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: stopping tablet replica
I20260926 14:50:34.673182 13214 tablet_replica.cc:330] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: stopping tablet replica
I20260926 14:50:34.673265 13215 raft_consensus.cc:2229] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:34.673265 13214 raft_consensus.cc:2229] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:34.673352 13214 raft_consensus.cc:2258] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:34.673352 13215 raft_consensus.cc:2258] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:34.673512 12683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:34.673660 12683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:34.673753 13215 ts_tablet_manager.cc:1905] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:34.673753 13214 ts_tablet_manager.cc:1905] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:34.675160 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 3ceb6c272b244e09abc064fb8ae0e378 (DEFAULT_TABLE table=test-table [id=4bed44ef6a254c749bab7b1505a72a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.675268 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ceb6c272b244e09abc064fb8ae0e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.675383 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 3ceb6c272b244e09abc064fb8ae0e378 (DEFAULT_TABLE table=test-table [id=4bed44ef6a254c749bab7b1505a72a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.675484 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ceb6c272b244e09abc064fb8ae0e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.675812 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 3ceb6c272b244e09abc064fb8ae0e378 (DEFAULT_TABLE table=test-table [id=4bed44ef6a254c749bab7b1505a72a7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:34.675915 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ceb6c272b244e09abc064fb8ae0e378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.676271 13216 tablet_replica.cc:330] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: stopping tablet replica
I20260926 14:50:34.676402 13216 raft_consensus.cc:2229] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:34.676541 13195 tablet_bootstrap.cc:492] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Bootstrap starting.
I20260926 14:50:34.676575 13215 ts_tablet_manager.cc:1918] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:34.676659 13215 log.cc:1196] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/e9d44e39b5474577838589820d5a7bbd
I20260926 14:50:34.676800 13216 raft_consensus.cc:2258] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:34.676936 13195 tablet_bootstrap.cc:654] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.676950 13215 ts_tablet_manager.cc:1939] T e9d44e39b5474577838589820d5a7bbd P 765383c591cb4d27bf165171f598d3f9: Deleting consensus metadata
I20260926 14:50:34.677321 13192 tablet_bootstrap.cc:492] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Bootstrap starting.
I20260926 14:50:34.677318 12667 catalog_manager.cc:4841] TS 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547): tablet e9d44e39b5474577838589820d5a7bbd (table test-table [id=837d7d271828409cab2135dcbc97252e]) successfully deleted
I20260926 14:50:34.677712 13192 tablet_bootstrap.cc:654] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.677949 13216 ts_tablet_manager.cc:1905] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:34.678428 13192 tablet_bootstrap.cc:492] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: No bootstrap required, opened a new log
I20260926 14:50:34.678483 13192 ts_tablet_manager.cc:1397] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:34.678520 13191 tablet_bootstrap.cc:492] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Bootstrap starting.
I20260926 14:50:34.678624 13192 raft_consensus.cc:348] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.678722 13192 raft_consensus.cc:374] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.678776 13192 raft_consensus.cc:729] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8938691ab2a4b0bbf1358d54c351c39, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.678824 13192 consensus_queue.cc:260] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.678929 13192 ts_tablet_manager.cc:1428] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:34.678938 13191 tablet_bootstrap.cc:654] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.679237 13214 ts_tablet_manager.cc:1918] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:34.679302 13214 log.cc:1196] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/e9d44e39b5474577838589820d5a7bbd
I20260926 14:50:34.679529 13195 tablet_bootstrap.cc:492] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: No bootstrap required, opened a new log
I20260926 14:50:34.679546 13214 ts_tablet_manager.cc:1939] T e9d44e39b5474577838589820d5a7bbd P 223ad00f8e774a7b81c42444192a3601: Deleting consensus metadata
I20260926 14:50:34.679577 13195 ts_tablet_manager.cc:1397] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:50:34.679711 13195 raft_consensus.cc:348] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.679775 13195 raft_consensus.cc:374] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.679795 13195 raft_consensus.cc:729] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.679834 13195 consensus_queue.cc:260] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.679922 12670 catalog_manager.cc:4841] TS 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479): tablet e9d44e39b5474577838589820d5a7bbd (table test-table [id=837d7d271828409cab2135dcbc97252e]) successfully deleted
I20260926 14:50:34.680146 13191 tablet_bootstrap.cc:492] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: No bootstrap required, opened a new log
I20260926 14:50:34.680154 13195 ts_tablet_manager.cc:1428] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:34.680200 13191 ts_tablet_manager.cc:1397] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:34.680344 13191 raft_consensus.cc:348] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.680424 13191 raft_consensus.cc:374] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.680450 13191 raft_consensus.cc:729] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.680490 13191 consensus_queue.cc:260] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.680593 13191 ts_tablet_manager.cc:1428] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.681128 13216 ts_tablet_manager.cc:1918] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:34.681237 13216 log.cc:1196] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/e9d44e39b5474577838589820d5a7bbd
I20260926 14:50:34.681488 13216 ts_tablet_manager.cc:1939] T e9d44e39b5474577838589820d5a7bbd P f8938691ab2a4b0bbf1358d54c351c39: Deleting consensus metadata
I20260926 14:50:34.681865 12669 catalog_manager.cc:4841] TS f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397): tablet e9d44e39b5474577838589820d5a7bbd (table test-table [id=837d7d271828409cab2135dcbc97252e]) successfully deleted
I20260926 14:50:34.832923 13199 raft_consensus.cc:482] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:34.833006 13199 raft_consensus.cc:504] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.833354 13199 leader_election.cc:288] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:34.836176 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.833576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:53005 (local address 127.5.15.130:41479)
0926 14:50:34.833623 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.833625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.833743 (+ 118us) server_negotiation.cc:407] Connection header received
0926 14:50:34.833796 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.833798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.833807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.833878 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.834026 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.834489 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.835469 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.835813 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.835896 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.835907 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.836063 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.836099 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 14:50:34.836189 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.833586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:55027 (local address 127.5.15.131:42397)
0926 14:50:34.833660 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.833662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.833781 (+ 119us) server_negotiation.cc:407] Connection header received
0926 14:50:34.833856 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.833857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.833864 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.833929 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:34.834074 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.834508 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.835472 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.835817 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.835906 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.835916 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.836087 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.836118 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:50:34.836246 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.833528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:41479 (local address 127.5.15.129:53005)
0926 14:50:34.833648 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.833660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.833739 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.833880 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.833880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.833882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.833956 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.833958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.834569 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.834570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.835377 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.835379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.835912 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.835913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.835983 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.835986 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.836057 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.836097 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":44,"thread_start_us":58,"threads_started":1}
I20260926 14:50:34.836221 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.833538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:42397 (local address 127.5.15.129:55027)
0926 14:50:34.833697 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.833724 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.833797 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.833931 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.833932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.833934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:34.834003 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.834009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.834587 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.834588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.835385 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.835387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.835925 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.835926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.836004 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.836007 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.836080 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.836118 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260926 14:50:34.836485 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb6c272b244e09abc064fb8ae0e378" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223ad00f8e774a7b81c42444192a3601" is_pre_election: true
I20260926 14:50:34.836582 12979 raft_consensus.cc:2454] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 0.
I20260926 14:50:34.836779 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb6c272b244e09abc064fb8ae0e378" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39" is_pre_election: true
I20260926 14:50:34.836831 12774 leader_election.cc:302] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:34.836930 13118 raft_consensus.cc:2454] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 0.
I20260926 14:50:34.836987 13199 raft_consensus.cc:2781] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:34.837045 13199 raft_consensus.cc:482] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:34.837070 13199 raft_consensus.cc:3037] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.837718 13199 raft_consensus.cc:504] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.837850 13199 leader_election.cc:288] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 election: Requested vote from peers 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:34.837989 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb6c272b244e09abc064fb8ae0e378" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223ad00f8e774a7b81c42444192a3601"
I20260926 14:50:34.838022 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ceb6c272b244e09abc064fb8ae0e378" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39"
I20260926 14:50:34.838093 13118 raft_consensus.cc:3037] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.838096 12979 raft_consensus.cc:3037] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:34.838666 12979 raft_consensus.cc:2454] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 1.
I20260926 14:50:34.838727 13118 raft_consensus.cc:2454] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 1.
I20260926 14:50:34.838871 12774 leader_election.cc:302] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:34.838974 13199 raft_consensus.cc:2781] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:34.839171 13199 raft_consensus.cc:686] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 1 LEADER]: Becoming Leader. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Running, Role: LEADER
I20260926 14:50:34.839252 13199 consensus_queue.cc:237] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.839962 12683 catalog_manager.cc:5495] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 765383c591cb4d27bf165171f598d3f9 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:50:34 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434233, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.190@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:34.889967 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.883596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.5.15.190:38433)
0926 14:50:34.883748 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:34.883749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:34.883758 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:34.883809 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:34.883810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:34.883819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:34.883877 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:34.884297 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.884826 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.885488 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.885665 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.885712 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.887609 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:34.887610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:34.887611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:34.887618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:34.889058 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.889370 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.889370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.889371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.889423 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:34.889579 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:34.889580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:34.889580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:34.889677 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:34.889726 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:34.889835 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 14:50:34.889901 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260926 14:50:34.890043 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:34.883514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38433 (local address 127.0.0.1:41716)
0926 14:50:34.883675 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:34.883687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:34.883753 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:34.883895 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:34.883895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:34.884190 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:34.884234 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.884236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.884887 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.884888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.885419 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:34.885420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:34.885724 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:34.885725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:34.885762 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:34.886219 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:34.886223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:34.887531 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:34.889126 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.889131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.889133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.889310 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.889479 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:34.889480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:34.889480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:34.889529 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:34.889730 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:34.889735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:34.889772 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:50:34.889821 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:50:34.889947 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":47,"thread_start_us":76,"threads_started":1}
I20260926 14:50:34.891873 12683 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:41716:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:50:34.892093 12683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:34.893461 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0b9ff4deb74463bda314f0970d9238 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=462eda9cc6b24b9ca77d5297ca17023f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:34.893504 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0b9ff4deb74463bda314f0970d9238 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=462eda9cc6b24b9ca77d5297ca17023f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:34.893496 12820 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0b9ff4deb74463bda314f0970d9238 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=462eda9cc6b24b9ca77d5297ca17023f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:50:34.893622 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0b9ff4deb74463bda314f0970d9238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.893630 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0b9ff4deb74463bda314f0970d9238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.893659 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0b9ff4deb74463bda314f0970d9238. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:34.894819 13195 tablet_bootstrap.cc:492] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: Bootstrap starting.
I20260926 14:50:34.895073 13191 tablet_bootstrap.cc:492] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: Bootstrap starting.
I20260926 14:50:34.895249 13195 tablet_bootstrap.cc:654] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.895476 13191 tablet_bootstrap.cc:654] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.895879 13192 tablet_bootstrap.cc:492] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: Bootstrap starting.
I20260926 14:50:34.896121 13191 tablet_bootstrap.cc:492] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: No bootstrap required, opened a new log
I20260926 14:50:34.896153 13195 tablet_bootstrap.cc:492] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: No bootstrap required, opened a new log
I20260926 14:50:34.896186 13191 ts_tablet_manager.cc:1397] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:34.896198 13195 ts_tablet_manager.cc:1397] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:34.896234 13192 tablet_bootstrap.cc:654] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:34.896340 13191 raft_consensus.cc:348] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.896347 13195 raft_consensus.cc:348] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.896427 13191 raft_consensus.cc:374] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.896431 13195 raft_consensus.cc:374] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.896457 13195 raft_consensus.cc:729] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.896456 13191 raft_consensus.cc:729] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.896507 13191 consensus_queue.cc:260] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.896510 13195 consensus_queue.cc:260] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.896603 13195 ts_tablet_manager.cc:1428] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:34.896618 13191 ts_tablet_manager.cc:1428] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.896720 13199 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:34.896791 13199 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:34.896912 13194 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:34.897004 13194 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:34.897176 13192 tablet_bootstrap.cc:492] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: No bootstrap required, opened a new log
I20260926 14:50:34.897249 13192 ts_tablet_manager.cc:1397] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:34.897363 13192 raft_consensus.cc:348] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.897433 13192 raft_consensus.cc:374] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:34.897452 13192 raft_consensus.cc:729] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8938691ab2a4b0bbf1358d54c351c39, State: Initialized, Role: FOLLOWER
I20260926 14:50:34.897503 13192 consensus_queue.cc:260] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:34.897620 13192 ts_tablet_manager.cc:1428] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:34.897653 13209 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:34.897719 13209 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:35.003146 13199 raft_consensus.cc:482] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:35.003252 13199 raft_consensus.cc:504] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.003424 13199 leader_election.cc:288] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:35.003649 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0b9ff4deb74463bda314f0970d9238" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223ad00f8e774a7b81c42444192a3601" is_pre_election: true
I20260926 14:50:35.003747 12979 raft_consensus.cc:2454] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 0.
I20260926 14:50:35.003751 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0b9ff4deb74463bda314f0970d9238" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39" is_pre_election: true
I20260926 14:50:35.003881 13118 raft_consensus.cc:2454] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 0.
I20260926 14:50:35.003890 12774 leader_election.cc:302] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:35.003973 13199 raft_consensus.cc:2781] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:35.004019 13199 raft_consensus.cc:482] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:35.004042 13199 raft_consensus.cc:3037] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.004666 13199 raft_consensus.cc:504] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.004760 13199 leader_election.cc:288] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 election: Requested vote from peers 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:35.004920 12979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0b9ff4deb74463bda314f0970d9238" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223ad00f8e774a7b81c42444192a3601"
I20260926 14:50:35.004927 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0b9ff4deb74463bda314f0970d9238" candidate_uuid: "765383c591cb4d27bf165171f598d3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39"
I20260926 14:50:35.005012 13118 raft_consensus.cc:3037] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.005015 12979 raft_consensus.cc:3037] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.005568 12979 raft_consensus.cc:2454] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 1.
I20260926 14:50:35.005640 13118 raft_consensus.cc:2454] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 765383c591cb4d27bf165171f598d3f9 in term 1.
I20260926 14:50:35.005874 12774 leader_election.cc:302] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:35.005980 13199 raft_consensus.cc:2781] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:35.006050 13199 raft_consensus.cc:686] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [term 1 LEADER]: Becoming Leader. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Running, Role: LEADER
I20260926 14:50:35.006106 13199 consensus_queue.cc:237] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.006438 13222 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: TxnStatusTablet state changed. Reason: New leader 765383c591cb4d27bf165171f598d3f9. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.006525 13222 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:35.006722 13235 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:50:35.006659 12683 catalog_manager.cc:5495] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 765383c591cb4d27bf165171f598d3f9 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:35.006872 13235 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:50:35.012346 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.009588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.0.0.1:53974)
0926 14:50:35.009685 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.009697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:35.009750 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:35.009953 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:35.009954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:35.009956 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:35.009998 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.010000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.010903 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.010904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.011471 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.011472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.011923 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.011924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.011960 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.011973 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.012208 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.012214 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:50:35.012248 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 14:50:35.012282 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 14:50:35.012348 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.009666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.5.15.129:40547)
0926 14:50:35.009717 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.009720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.009729 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:35.009832 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.009833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.009853 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.009931 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:35.010070 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.010830 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.011552 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.011848 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.011931 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.012038 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.012146 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.012217 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.012254 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.012275 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:50:35.013281 12979 raft_consensus.cc:1261] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Refusing update from remote peer 765383c591cb4d27bf165171f598d3f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.013341 13118 raft_consensus.cc:1261] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Refusing update from remote peer 765383c591cb4d27bf165171f598d3f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.013566 13222 consensus_queue.cc:1035] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.013688 13199 consensus_queue.cc:1035] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.014268 13209 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: TxnStatusTablet state changed. Reason: New leader 765383c591cb4d27bf165171f598d3f9. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.014356 13209 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:35.014649 13194 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: TxnStatusTablet state changed. Reason: New leader 765383c591cb4d27bf165171f598d3f9. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.014731 13194 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:35.015172 13222 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.015249 13222 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:35.015297 13222 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.015345 13222 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 765383c591cb4d27bf165171f598d3f9: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:50:35.015748 13194 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.015830 13194 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P 223ad00f8e774a7b81c42444192a3601: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:35.015940 13209 tablet_replica.cc:439] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "765383c591cb4d27bf165171f598d3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } } }
I20260926 14:50:35.016024 13209 tablet_replica.cc:442] T 8b0b9ff4deb74463bda314f0970d9238 P f8938691ab2a4b0bbf1358d54c351c39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:50:35.019404 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.017194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.5.15.129:40547)
0926 14:50:35.017237 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.017238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.017245 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:35.017297 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.017298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.017305 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.017370 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:35.017479 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.017988 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.018691 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.018933 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.019009 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.019203 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.019230 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.019271 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.019333 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.019348 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:50:35.024183 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.021728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:51485 (local address 127.5.15.129:40547)
0926 14:50:35.021763 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.021765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.021862 (+ 97us) server_negotiation.cc:407] Connection header received
0926 14:50:35.021921 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.021921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.021927 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.021970 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:35.022080 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.022462 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.023597 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.023755 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.023803 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.023992 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.024011 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.024067 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.024118 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.024130 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:50:35.024271 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.021661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.5.15.129:51485)
0926 14:50:35.021797 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.021806 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:35.021861 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:35.021978 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:35.021979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:35.021980 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:35.022030 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.022031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.022524 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.022525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.023527 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.023529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.023840 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.023841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.023910 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.023922 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.024069 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.024070 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:50:35.024116 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:50:35.024151 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260926 14:50:35.025853 13118 raft_consensus.cc:1261] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Refusing update from remote peer 765383c591cb4d27bf165171f598d3f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.025899 12979 raft_consensus.cc:1261] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Refusing update from remote peer 765383c591cb4d27bf165171f598d3f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.026038 13199 consensus_queue.cc:1035] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.026147 13199 consensus_queue.cc:1035] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.037019 12683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:35.037106 12683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:35.037670 12683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Sending DeleteTablet for 3 replicas of tablet 3ceb6c272b244e09abc064fb8ae0e378
I20260926 14:50:35.037896 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ceb6c272b244e09abc064fb8ae0e378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:46112
I20260926 14:50:35.037999 13214 tablet_replica.cc:330] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: stopping tablet replica
I20260926 14:50:35.037978 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ceb6c272b244e09abc064fb8ae0e378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:47934
I20260926 14:50:35.038046 13214 raft_consensus.cc:2229] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:35.038082 13214 raft_consensus.cc:2258] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.038066 13216 tablet_replica.cc:330] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: stopping tablet replica
I20260926 14:50:35.038131 13216 raft_consensus.cc:2229] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:35.038163 13216 raft_consensus.cc:2258] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.038302 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ceb6c272b244e09abc064fb8ae0e378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:53966
I20260926 14:50:35.038408 13214 ts_tablet_manager.cc:1905] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.039912 13215 tablet_replica.cc:330] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: stopping tablet replica
I20260926 14:50:35.039969 13215 raft_consensus.cc:2229] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:35.040053 13215 raft_consensus.cc:2258] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.040133 13214 ts_tablet_manager.cc:1918] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:35.040184 13214 log.cc:1196] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/3ceb6c272b244e09abc064fb8ae0e378
I20260926 14:50:35.040385 13214 ts_tablet_manager.cc:1939] T 3ceb6c272b244e09abc064fb8ae0e378 P 223ad00f8e774a7b81c42444192a3601: Deleting consensus metadata
I20260926 14:50:35.040704 12670 catalog_manager.cc:4841] TS 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479): tablet 3ceb6c272b244e09abc064fb8ae0e378 (table test-table [id=4bed44ef6a254c749bab7b1505a72a7b]) successfully deleted
I20260926 14:50:35.041085 13215 ts_tablet_manager.cc:1905] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.042377 13215 ts_tablet_manager.cc:1918] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:35.042448 13215 log.cc:1196] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/3ceb6c272b244e09abc064fb8ae0e378
I20260926 14:50:35.042829 13215 ts_tablet_manager.cc:1939] T 3ceb6c272b244e09abc064fb8ae0e378 P 765383c591cb4d27bf165171f598d3f9: Deleting consensus metadata
I20260926 14:50:35.043183 12667 catalog_manager.cc:4841] TS 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547): tablet 3ceb6c272b244e09abc064fb8ae0e378 (table test-table [id=4bed44ef6a254c749bab7b1505a72a7b]) successfully deleted
I20260926 14:50:35.043219 13216 ts_tablet_manager.cc:1905] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.044517 13216 ts_tablet_manager.cc:1918] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:35.044602 13216 log.cc:1196] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/3ceb6c272b244e09abc064fb8ae0e378
I20260926 14:50:35.044828 13216 ts_tablet_manager.cc:1939] T 3ceb6c272b244e09abc064fb8ae0e378 P f8938691ab2a4b0bbf1358d54c351c39: Deleting consensus metadata
I20260926 14:50:35.045114 12669 catalog_manager.cc:4841] TS f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397): tablet 3ceb6c272b244e09abc064fb8ae0e378 (table test-table [id=4bed44ef6a254c749bab7b1505a72a7b]) successfully deleted
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[12638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:35.068322 13230 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:35.062663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.5.15.190:38433)
0926 14:50:35.062759 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.062761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.062905 (+ 144us) server_negotiation.cc:407] Connection header received
0926 14:50:35.062989 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.062990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.063010 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.063082 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:35.063682 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.064230 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.065002 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.065211 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.065296 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.068242 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41732: BlockingRecv error: recv got EOF from 127.0.0.1:41732 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[12638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM
I20260926 14:50:35.076156 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.069119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.5.15.190:38433)
0926 14:50:35.069155 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.069157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.069266 (+ 109us) server_negotiation.cc:407] Connection header received
0926 14:50:35.069336 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.069337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.069349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.069405 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:35.069898 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.070455 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.071206 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.071471 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.071526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.073818 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:35.073819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:35.073819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:35.073825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:35.075149 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:35.075514 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:35.075514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:35.075515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:35.075568 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:35.075750 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:35.075751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:35.075751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:35.075839 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:35.075906 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.076037 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.076097 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:35.077600 12683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:35.077740 12683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:35.079164 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 42770900b119433a88927f365f8b406d (DEFAULT_TABLE table=test-table [id=901bee2a6e744759bb09dbb11162a9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:35.079216 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 42770900b119433a88927f365f8b406d (DEFAULT_TABLE table=test-table [id=901bee2a6e744759bb09dbb11162a9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:35.079247 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 42770900b119433a88927f365f8b406d (DEFAULT_TABLE table=test-table [id=901bee2a6e744759bb09dbb11162a9ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:35.079294 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42770900b119433a88927f365f8b406d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:35.079308 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42770900b119433a88927f365f8b406d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:35.079365 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42770900b119433a88927f365f8b406d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:35.080576 13191 tablet_bootstrap.cc:492] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Bootstrap starting.
I20260926 14:50:35.080593 13195 tablet_bootstrap.cc:492] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Bootstrap starting.
I20260926 14:50:35.080891 13195 tablet_bootstrap.cc:654] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:35.080960 13191 tablet_bootstrap.cc:654] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:35.081245 13192 tablet_bootstrap.cc:492] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Bootstrap starting.
I20260926 14:50:35.081442 13195 tablet_bootstrap.cc:492] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: No bootstrap required, opened a new log
I20260926 14:50:35.081473 13195 ts_tablet_manager.cc:1397] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:35.081530 13192 tablet_bootstrap.cc:654] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:35.081573 13195 raft_consensus.cc:348] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.081619 13195 raft_consensus.cc:374] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:35.081630 13195 raft_consensus.cc:729] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Initialized, Role: FOLLOWER
I20260926 14:50:35.081689 13195 consensus_queue.cc:260] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.081808 13195 ts_tablet_manager.cc:1428] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:35.082378 13191 tablet_bootstrap.cc:492] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: No bootstrap required, opened a new log
I20260926 14:50:35.082432 13191 ts_tablet_manager.cc:1397] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:50:35.082573 13191 raft_consensus.cc:348] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.082636 13191 raft_consensus.cc:374] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:35.082654 13191 raft_consensus.cc:729] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Initialized, Role: FOLLOWER
I20260926 14:50:35.082691 13191 consensus_queue.cc:260] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.082818 13191 ts_tablet_manager.cc:1428] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:35.083218 13192 tablet_bootstrap.cc:492] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: No bootstrap required, opened a new log
I20260926 14:50:35.083271 13192 ts_tablet_manager.cc:1397] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:35.083367 13192 raft_consensus.cc:348] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.083434 13192 raft_consensus.cc:374] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:35.083453 13192 raft_consensus.cc:729] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8938691ab2a4b0bbf1358d54c351c39, State: Initialized, Role: FOLLOWER
I20260926 14:50:35.083521 13192 consensus_queue.cc:260] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.083606 13192 ts_tablet_manager.cc:1428] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:35.113251 13194 raft_consensus.cc:482] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:35.113358 13194 raft_consensus.cc:504] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.113768 13194 leader_election.cc:288] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:35.117749 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.113888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.5.15.130:37243)
0926 14:50:35.114248 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.114259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:35.114322 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:35.114514 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:35.114515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:35.114519 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:35.114597 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.114599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.115295 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.115296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.116900 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.116902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.117459 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.117460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.117530 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.117534 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:35.117614 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:50:35.117663 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":80,"threads_started":1}
I20260926 14:50:35.117781 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.114094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:37243 (local address 127.5.15.129:40547)
0926 14:50:35.114163 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.114166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.114333 (+ 167us) server_negotiation.cc:407] Connection header received
0926 14:50:35.114371 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.114372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.114382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.114450 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:35.114695 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.115212 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.116997 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.117347 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.117438 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.117448 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.117651 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.117692 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:50:35.117810 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.114134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:55195 (local address 127.5.15.131:42397)
0926 14:50:35.114176 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.114178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.114446 (+ 268us) server_negotiation.cc:407] Connection header received
0926 14:50:35.114551 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.114552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.114563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.114607 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:35.114793 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.115496 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.116901 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.117451 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.117546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.117556 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.117697 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.117736 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:50:35.118024 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.113922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:42397 (local address 127.5.15.130:55195)
0926 14:50:35.114371 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.114381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:35.114468 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:35.114618 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:35.114619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:35.114621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:35.114697 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.114699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.115600 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.115601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.116788 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.116790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.117546 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.117547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.117616 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.117619 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:35.117884 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 14:50:35.117942 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":3,"thread_start_us":48,"threads_started":1}
I20260926 14:50:35.118013 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42770900b119433a88927f365f8b406d" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765383c591cb4d27bf165171f598d3f9" is_pre_election: true
I20260926 14:50:35.118108 12840 raft_consensus.cc:2454] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 0.
I20260926 14:50:35.118418 12911 leader_election.cc:302] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:35.118549 13194 raft_consensus.cc:2781] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:35.118580 13194 raft_consensus.cc:482] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:35.118605 13194 raft_consensus.cc:3037] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.118682 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42770900b119433a88927f365f8b406d" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39" is_pre_election: true
I20260926 14:50:35.118799 13118 raft_consensus.cc:2454] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 0.
I20260926 14:50:35.119401 13194 raft_consensus.cc:504] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.119561 13194 leader_election.cc:288] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 election: Requested vote from peers 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:35.119711 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42770900b119433a88927f365f8b406d" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765383c591cb4d27bf165171f598d3f9"
I20260926 14:50:35.119735 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42770900b119433a88927f365f8b406d" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39"
I20260926 14:50:35.119782 12840 raft_consensus.cc:3037] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.119802 13118 raft_consensus.cc:3037] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.120376 12840 raft_consensus.cc:2454] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 1.
I20260926 14:50:35.120551 12911 leader_election.cc:302] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:35.120622 13118 raft_consensus.cc:2454] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 1.
I20260926 14:50:35.120633 13194 raft_consensus.cc:2781] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:35.120761 13194 raft_consensus.cc:686] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 1 LEADER]: Becoming Leader. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Running, Role: LEADER
I20260926 14:50:35.120831 13194 consensus_queue.cc:237] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.121454 12682 catalog_manager.cc:5495] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 reported cstate change: term changed from 0 to 1, leader changed from <none> to 223ad00f8e774a7b81c42444192a3601 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "223ad00f8e774a7b81c42444192a3601" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:35.133860 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.130476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.5.15.130:41479)
0926 14:50:35.130513 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.130515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.131036 (+ 521us) server_negotiation.cc:407] Connection header received
0926 14:50:35.131187 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.131188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.131206 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.131286 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:35.131555 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.132154 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.133184 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.133359 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.133405 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.133592 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.133675 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.133721 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.133799 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.133813 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:50:35.134684 13118 raft_consensus.cc:1261] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Refusing update from remote peer 223ad00f8e774a7b81c42444192a3601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.134780 12840 raft_consensus.cc:1261] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Refusing update from remote peer 223ad00f8e774a7b81c42444192a3601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.135102 13194 consensus_queue.cc:1035] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.135239 13264 consensus_queue.cc:1035] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.136644 13206 mvcc.cc:205] Tried to move back new op lower bound from 7333618627109904384 to 7333618627055460352. Current Snapshot: MvccSnapshot[applied={T|T < 7333618627109904384}]
I20260926 14:50:35.138942 12683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:35.139030 12683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:35.139648 12683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Sending DeleteTablet for 3 replicas of tablet 42770900b119433a88927f365f8b406d
I20260926 14:50:35.139904 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 42770900b119433a88927f365f8b406d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:53966
I20260926 14:50:35.140002 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 42770900b119433a88927f365f8b406d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:47934
I20260926 14:50:35.140019 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 42770900b119433a88927f365f8b406d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:46112
I20260926 14:50:35.140080 13215 tablet_replica.cc:330] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: stopping tablet replica
I20260926 14:50:35.140097 13214 tablet_replica.cc:330] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: stopping tablet replica
I20260926 14:50:35.140125 13215 raft_consensus.cc:2229] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:35.140141 13214 raft_consensus.cc:2229] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:35.140163 13215 raft_consensus.cc:2258] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.140208 13214 raft_consensus.cc:2258] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.140224 13216 tablet_replica.cc:330] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: stopping tablet replica
I20260926 14:50:35.140268 13216 raft_consensus.cc:2229] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:35.140302 13216 raft_consensus.cc:2258] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.140683 12683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:35.140811 12683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:35.140875 13215 ts_tablet_manager.cc:1905] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.141988 13099 tablet_service.cc:1467] Processing CreateTablet for tablet 7d874f10960e43ab92f019625209d162 (DEFAULT_TABLE table=test-table [id=ed761eb4cfa34ad6b123f1643fec1b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:35.142091 13099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d874f10960e43ab92f019625209d162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:35.142372 13215 ts_tablet_manager.cc:1918] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:35.142419 13215 log.cc:1196] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/42770900b119433a88927f365f8b406d
I20260926 14:50:35.142642 13215 ts_tablet_manager.cc:1939] T 42770900b119433a88927f365f8b406d P 765383c591cb4d27bf165171f598d3f9: Deleting consensus metadata
I20260926 14:50:35.143201 13216 ts_tablet_manager.cc:1905] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.143489 13192 tablet_bootstrap.cc:492] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Bootstrap starting.
I20260926 14:50:35.143874 13192 tablet_bootstrap.cc:654] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:35.144178 12819 tablet_service.cc:1467] Processing CreateTablet for tablet 7d874f10960e43ab92f019625209d162 (DEFAULT_TABLE table=test-table [id=ed761eb4cfa34ad6b123f1643fec1b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:35.144274 12819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d874f10960e43ab92f019625209d162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:35.144490 13216 ts_tablet_manager.cc:1918] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:35.144536 13216 log.cc:1196] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/42770900b119433a88927f365f8b406d
I20260926 14:50:35.144734 13216 ts_tablet_manager.cc:1939] T 42770900b119433a88927f365f8b406d P f8938691ab2a4b0bbf1358d54c351c39: Deleting consensus metadata
I20260926 14:50:35.145318 13192 tablet_bootstrap.cc:492] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: No bootstrap required, opened a new log
I20260926 14:50:35.145366 13192 ts_tablet_manager.cc:1397] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:50:35.145459 13192 raft_consensus.cc:348] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.145515 13192 raft_consensus.cc:374] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:35.145540 13192 raft_consensus.cc:729] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8938691ab2a4b0bbf1358d54c351c39, State: Initialized, Role: FOLLOWER
I20260926 14:50:35.145551 13195 tablet_bootstrap.cc:492] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9: Bootstrap starting.
I20260926 14:50:35.145582 13192 consensus_queue.cc:260] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.145660 13192 ts_tablet_manager.cc:1428] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:35.145910 13195 tablet_bootstrap.cc:654] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:35.146556 12960 tablet_service.cc:1467] Processing CreateTablet for tablet 7d874f10960e43ab92f019625209d162 (DEFAULT_TABLE table=test-table [id=ed761eb4cfa34ad6b123f1643fec1b1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:35.146661 12960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d874f10960e43ab92f019625209d162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:35.146982 13195 tablet_bootstrap.cc:492] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9: No bootstrap required, opened a new log
I20260926 14:50:35.147030 13195 ts_tablet_manager.cc:1397] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:35.147190 13195 raft_consensus.cc:348] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.147245 13195 raft_consensus.cc:374] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:35.147264 13195 raft_consensus.cc:729] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765383c591cb4d27bf165171f598d3f9, State: Initialized, Role: FOLLOWER
I20260926 14:50:35.147303 13195 consensus_queue.cc:260] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.147378 13195 ts_tablet_manager.cc:1428] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:35.148037 13191 tablet_bootstrap.cc:492] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Bootstrap starting.
I20260926 14:50:35.148275 12669 catalog_manager.cc:4841] TS f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397): tablet 42770900b119433a88927f365f8b406d (table test-table [id=901bee2a6e744759bb09dbb11162a9ed]) successfully deleted
I20260926 14:50:35.148423 13191 tablet_bootstrap.cc:654] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:35.148550 12667 catalog_manager.cc:4841] TS 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547): tablet 42770900b119433a88927f365f8b406d (table test-table [id=901bee2a6e744759bb09dbb11162a9ed]) successfully deleted
I20260926 14:50:35.148762 13214 ts_tablet_manager.cc:1905] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.149955 13214 ts_tablet_manager.cc:1918] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:35.150017 13214 log.cc:1196] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/42770900b119433a88927f365f8b406d
I20260926 14:50:35.150220 13214 ts_tablet_manager.cc:1939] T 42770900b119433a88927f365f8b406d P 223ad00f8e774a7b81c42444192a3601: Deleting consensus metadata
I20260926 14:50:35.150596 12670 catalog_manager.cc:4841] TS 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479): tablet 42770900b119433a88927f365f8b406d (table test-table [id=901bee2a6e744759bb09dbb11162a9ed]) successfully deleted
I20260926 14:50:35.150672 13191 tablet_bootstrap.cc:492] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: No bootstrap required, opened a new log
I20260926 14:50:35.150717 13191 ts_tablet_manager.cc:1397] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 14:50:35.150856 13191 raft_consensus.cc:348] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.150918 13191 raft_consensus.cc:374] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:35.150936 13191 raft_consensus.cc:729] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Initialized, Role: FOLLOWER
I20260926 14:50:35.150970 13191 consensus_queue.cc:260] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.151039 13191 ts_tablet_manager.cc:1428] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:35.238479 13269 raft_consensus.cc:482] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:35.238600 13269 raft_consensus.cc:504] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.238797 13269 leader_election.cc:288] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:35.239001 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d874f10960e43ab92f019625209d162" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765383c591cb4d27bf165171f598d3f9" is_pre_election: true
I20260926 14:50:35.239099 12840 raft_consensus.cc:2454] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 0.
I20260926 14:50:35.239041 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d874f10960e43ab92f019625209d162" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39" is_pre_election: true
I20260926 14:50:35.239154 13118 raft_consensus.cc:2454] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 0.
I20260926 14:50:35.239285 12911 leader_election.cc:302] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:35.239464 13269 raft_consensus.cc:2781] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:35.239495 13269 raft_consensus.cc:482] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:35.239513 13269 raft_consensus.cc:3037] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.240327 13269 raft_consensus.cc:504] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.240598 12840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d874f10960e43ab92f019625209d162" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765383c591cb4d27bf165171f598d3f9"
I20260926 14:50:35.240669 12840 raft_consensus.cc:3037] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.240688 13118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d874f10960e43ab92f019625209d162" candidate_uuid: "223ad00f8e774a7b81c42444192a3601" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8938691ab2a4b0bbf1358d54c351c39"
I20260926 14:50:35.240760 13118 raft_consensus.cc:3037] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:35.241417 12840 raft_consensus.cc:2454] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 1.
I20260926 14:50:35.241478 13118 raft_consensus.cc:2454] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 223ad00f8e774a7b81c42444192a3601 in term 1.
I20260926 14:50:35.241601 12911 leader_election.cc:302] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 223ad00f8e774a7b81c42444192a3601, 765383c591cb4d27bf165171f598d3f9; no voters:
I20260926 14:50:35.241660 13269 leader_election.cc:288] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [CANDIDATE]: Term 1 election: Requested vote from peers 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547), f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397)
I20260926 14:50:35.241704 13264 raft_consensus.cc:2781] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:35.241747 13264 raft_consensus.cc:686] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 1 LEADER]: Becoming Leader. State: Replica: 223ad00f8e774a7b81c42444192a3601, State: Running, Role: LEADER
I20260926 14:50:35.241788 13264 consensus_queue.cc:237] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } }
I20260926 14:50:35.242272 12681 catalog_manager.cc:5495] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 reported cstate change: term changed from 0 to 1, leader changed from <none> to 223ad00f8e774a7b81c42444192a3601 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "223ad00f8e774a7b81c42444192a3601" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "223ad00f8e774a7b81c42444192a3601" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 41479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:35.270364 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.267799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.5.15.130:58205)
0926 14:50:35.267960 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.267971 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:35.268039 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:35.268220 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:35.268221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:35.268223 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:35.268275 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.268277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.268895 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.268896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.269499 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.269501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.269891 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.269892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.269946 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.269967 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.270190 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.270192 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:35.270237 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:50:35.270289 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260926 14:50:35.270332 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.267900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:58205 (local address 127.5.15.129:40547)
0926 14:50:35.267954 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.267956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.268022 (+ 66us) server_negotiation.cc:407] Connection header received
0926 14:50:35.268100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.268101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.268111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.268187 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:35.268322 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.268810 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.269595 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.269821 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.269883 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.270064 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.270102 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.270185 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.270241 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.270262 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:50:35.272131 13118 raft_consensus.cc:1261] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Refusing update from remote peer 223ad00f8e774a7b81c42444192a3601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.272138 12840 raft_consensus.cc:1261] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Refusing update from remote peer 223ad00f8e774a7b81c42444192a3601: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:35.272356 13269 consensus_queue.cc:1035] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8938691ab2a4b0bbf1358d54c351c39" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.272476 13264 consensus_queue.cc:1035] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [LEADER]: Connected to new peer: Peer: permanent_uuid: "765383c591cb4d27bf165171f598d3f9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 40547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:35.278808 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.276562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:41479 (local address 127.5.15.129:40349)
0926 14:50:35.276643 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.276655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:35.276694 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:35.276832 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:35.276833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:35.276834 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:50:35.276874 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.276876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.277506 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.277507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.278047 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:35.278048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.278346 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.278347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:35.278382 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.278387 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.278603 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.278606 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:35.278651 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:50:35.278696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:50:35.278822 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.276658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:40349 (local address 127.5.15.130:41479)
0926 14:50:35.276704 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:35.276706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:35.276712 (+ 6us) server_negotiation.cc:407] Connection header received
0926 14:50:35.276731 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:35.276731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:35.276757 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:35.276826 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:35.276938 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.277444 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.278116 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:35.278286 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:35.278333 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:35.278473 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:35.278518 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:35.278566 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:35.278713 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:50:35.278736 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:35.285072 12683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:35.285149 12683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:35.285698 12683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c9292afadb426ba575c1210f483195: Sending DeleteTablet for 3 replicas of tablet 7d874f10960e43ab92f019625209d162
I20260926 14:50:35.285919 12960 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d874f10960e43ab92f019625209d162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:46112
I20260926 14:50:35.285974 13099 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d874f10960e43ab92f019625209d162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:47934
I20260926 14:50:35.286043 13214 tablet_replica.cc:330] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: stopping tablet replica
I20260926 14:50:35.286074 13216 tablet_replica.cc:330] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: stopping tablet replica
I20260926 14:50:35.286099 13214 raft_consensus.cc:2229] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:50:35.286109 12819 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d874f10960e43ab92f019625209d162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:35 UTC) from {username='oryx', principal='oryx/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:53966
I20260926 14:50:35.286134 13216 raft_consensus.cc:2229] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:35.286163 13216 raft_consensus.cc:2258] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.286161 13214 raft_consensus.cc:2258] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.286175 13215 tablet_replica.cc:330] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9: stopping tablet replica
I20260926 14:50:35.286214 13215 raft_consensus.cc:2229] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:35.286247 13215 raft_consensus.cc:2258] T 7d874f10960e43ab92f019625209d162 P 765383c591cb4d27bf165171f598d3f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:35.286468 13216 ts_tablet_manager.cc:1905] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.286520 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 12755
I20260926 14:50:35.287959 13216 ts_tablet_manager.cc:1918] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:35.288043 13216 log.cc:1196] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/7d874f10960e43ab92f019625209d162
I20260926 14:50:35.288247 13216 ts_tablet_manager.cc:1939] T 7d874f10960e43ab92f019625209d162 P f8938691ab2a4b0bbf1358d54c351c39: Deleting consensus metadata
I20260926 14:50:35.288555 12669 catalog_manager.cc:4841] TS f8938691ab2a4b0bbf1358d54c351c39 (127.5.15.131:42397): tablet 7d874f10960e43ab92f019625209d162 (table test-table [id=ed761eb4cfa34ad6b123f1643fec1b1f]) successfully deleted
I20260926 14:50:35.291018 13214 ts_tablet_manager.cc:1905] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:35.292698 13214 ts_tablet_manager.cc:1918] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:50:35.292762 13214 log.cc:1196] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/7d874f10960e43ab92f019625209d162
I20260926 14:50:35.292963 13214 ts_tablet_manager.cc:1939] T 7d874f10960e43ab92f019625209d162 P 223ad00f8e774a7b81c42444192a3601: Deleting consensus metadata
I20260926 14:50:35.293293 12670 catalog_manager.cc:4841] TS 223ad00f8e774a7b81c42444192a3601 (127.5.15.130:41479): tablet 7d874f10960e43ab92f019625209d162 (table test-table [id=ed761eb4cfa34ad6b123f1643fec1b1f]) successfully deleted
W20260926 14:50:35.294899 12667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:40547 (error 108)
I20260926 14:50:35.295157 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 12895
I20260926 14:50:35.295360 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:35.295189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:40547 (local address 127.0.0.1:53982)
0926 14:50:35.295278 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:35.295317 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:40547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":51,"threads_started":1}
W20260926 14:50:35.296193 12667 catalog_manager.cc:4584] TS 765383c591cb4d27bf165171f598d3f9 (127.5.15.129:40547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d874f10960e43ab92f019625209d162: Network error: Client connection negotiation failed: client connection to 127.5.15.129:40547: connect: Connection refused (error 111)
I20260926 14:50:35.301523 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13035
I20260926 14:50:35.306588 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 12651
2026-09-26T14:50:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2685 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[13284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[13284](info): set up 2 sockets
Sep 26 14:50:35 dist-test-slave-s4p1 krb5kdc[13284](info): commencing operation
krb5kdc: starting...
W20260926 14:50:36.315299 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:36 dist-test-slave-s4p1 krb5kdc[13284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:36.428920 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:34113
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:34113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:36.511458 13297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:36.511654 13297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:36.511677 13297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:36.512890 13297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:36.512928 13297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:36.512943 13297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:36.512955 13297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:36.513000 13297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:36.514447 13297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:34113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:34113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:36.514665 13297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:36.514931 13297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:36.519500 13302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:36.519564 13303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:36.519609 13297 server_base.cc:1034] running on GCE node
W20260926 14:50:36.519493 13305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:36.519877 13297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:36.520154 13297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:36.521292 13297 hybrid_clock.cc:648] HybridClock initialized: now 1790434236521284 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T14:50:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1218 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:36 dist-test-slave-s4p1 krb5kdc[13310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:36 dist-test-slave-s4p1 krb5kdc[13310](info): set up 2 sockets
Sep 26 14:50:36 dist-test-slave-s4p1 krb5kdc[13310](info): commencing operation
krb5kdc: starting...
W20260926 14:50:37.543377 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:37 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:37.658928 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38653
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:37.735387 13323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:37.735567 13323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:37.735589 13323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:37.736694 13323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:37.736745 13323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:37.736759 13323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:37.736809 13323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:37.736835 13323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:37.738416 13323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:37.738660 13323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:37.738897 13323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:37.743494 13329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:37.743474 13328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:37.743482 13331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:37.743547 13323 server_base.cc:1034] running on GCE node
I20260926 14:50:37.743885 13323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:37.744174 13323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:37.745309 13323 hybrid_clock.cc:648] HybridClock initialized: now 1790434237745295 us; error 30 us; skew 500 ppm
Sep 26 14:50:37 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:37.747814 13323 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:37.748245 13323 webserver.cc:460] Webserver started at http://127.5.15.190:39195/ using document root <none> and password file <none>
I20260926 14:50:37.748467 13323 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:37.748513 13323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:37.748660 13323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:37.749624 13323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "320a17ba06f1405896ca625f08eb8290"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "6ede322046ee24fd1c1d6a2ff344da5d"
server_key_iv: "89fe97753c7cffcdd3ef04bab440e7e4"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.749964 13323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "320a17ba06f1405896ca625f08eb8290"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "6ede322046ee24fd1c1d6a2ff344da5d"
server_key_iv: "89fe97753c7cffcdd3ef04bab440e7e4"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.751240 13323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:37.751904 13338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:37.752103 13323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:37.752209 13323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "320a17ba06f1405896ca625f08eb8290"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "6ede322046ee24fd1c1d6a2ff344da5d"
server_key_iv: "89fe97753c7cffcdd3ef04bab440e7e4"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.752288 13323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:37.760354 13323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:37.762029 13323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:37.762180 13323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:37.765774 13323 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:38653
I20260926 14:50:37.765802 13390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:38653 every 8 connection(s)
I20260926 14:50:37.766104 13323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:37.766885 13391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:37.768908 13391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Bootstrap starting.
I20260926 14:50:37.769464 13391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:37.769747 13391 log.cc:824] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:37.770377 13391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: No bootstrap required, opened a new log
I20260926 14:50:37.771893 13391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:37.772015 13391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:37.772060 13391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320a17ba06f1405896ca625f08eb8290, State: Initialized, Role: FOLLOWER
I20260926 14:50:37.772202 13391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:37.772258 13391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:37.772297 13391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:37.772348 13391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:37.772815 13391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:37.772922 13391 leader_election.cc:302] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320a17ba06f1405896ca625f08eb8290; no voters:
I20260926 14:50:37.773098 13391 leader_election.cc:288] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:37.773135 13394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:37.773275 13394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 LEADER]: Becoming Leader. State: Replica: 320a17ba06f1405896ca625f08eb8290, State: Running, Role: LEADER
I20260926 14:50:37.773360 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13323
I20260926 14:50:37.773381 13391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:37.773471 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:37.773433 13394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:37.773663 5182 external_mini_cluster.cc:1428] Setting key 44f4180a6cc40ed736374005d96ef077
I20260926 14:50:37.774080 13395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320a17ba06f1405896ca625f08eb8290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } } }
I20260926 14:50:37.774173 13395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:37.774474 13396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320a17ba06f1405896ca625f08eb8290. Latest consensus state: current_term: 1 leader_uuid: "320a17ba06f1405896ca625f08eb8290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } } }
I20260926 14:50:37.774540 13396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: This master's current role is: LEADER
W20260926 14:50:37.775174 13412 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:50:37.775444 13412 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:50:37.775507 13404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:37.775664 13404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:37.777058 13404 catalog_manager.cc:1347] Generated new cluster ID: 67a9947448584626836adadafdca637c
I20260926 14:50:37.777117 13404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:37 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:37.785440 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:37.774300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.5.15.190:38653)
0926 14:50:37.774472 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:37.774481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:37.774947 (+ 466us) server_negotiation.cc:407] Connection header received
0926 14:50:37.775103 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:37.775111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:37.775167 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:37.775316 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:37.776058 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.777406 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.778518 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.779162 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.779246 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:37.781999 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:37.782005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:37.782005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:37.782019 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:37.783916 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:37.784290 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:37.784293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:37.784293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:37.784403 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:37.784592 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:37.784593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:37.784593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:37.784887 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:37.785044 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:37.785119 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:50:37.785244 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260926 14:50:37.799000 13404 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:37.799664 13404 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:37.806118 13404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Generated new TSK 0
I20260926 14:50:37.806294 13404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:37.837349 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38653
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:37.915426 13417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:37.915616 13417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:37.915639 13417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:37.916770 13417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:37.916810 13417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:37.916863 13417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:37.918259 13417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:37.918498 13417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:37.918730 13417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:37.923482 13422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:37.923499 13423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:37.923516 13425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:37.923820 13417 server_base.cc:1034] running on GCE node
I20260926 14:50:37.924033 13417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:37.924271 13417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:37.925434 13417 hybrid_clock.cc:648] HybridClock initialized: now 1790434237925422 us; error 25 us; skew 500 ppm
Sep 26 14:50:37 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:37.927870 13417 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:37.928277 13417 webserver.cc:460] Webserver started at http://127.5.15.129:33465/ using document root <none> and password file <none>
I20260926 14:50:37.928519 13417 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:37.928586 13417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:37.928732 13417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:37.929713 13417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "2fd896d774d846419edabc0214f70943"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "0b0f7f6b1c87ac24a4e7c33c48e37b5a"
server_key_iv: "c14b5663fa94570f50c6e5fdd391a9f1"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.930086 13417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "2fd896d774d846419edabc0214f70943"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "0b0f7f6b1c87ac24a4e7c33c48e37b5a"
server_key_iv: "c14b5663fa94570f50c6e5fdd391a9f1"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.931492 13417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:37.932159 13432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:37.932322 13417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:37.932417 13417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "2fd896d774d846419edabc0214f70943"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "0b0f7f6b1c87ac24a4e7c33c48e37b5a"
server_key_iv: "c14b5663fa94570f50c6e5fdd391a9f1"
server_key_version: "encryptionkey@0"
I20260926 14:50:37.932497 13417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:37.939180 13417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:37.940937 13417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:37.941088 13417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:37.941285 13417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:37.941561 13417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:37.941613 13417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:37.941649 13417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:37.941677 13417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:37.947371 13417 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:38681
I20260926 14:50:37.947414 13545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:38681 every 8 connection(s)
I20260926 14:50:37.947692 13417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:37.951610 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13417
I20260926 14:50:37.951735 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:37.951910 5182 external_mini_cluster.cc:1428] Setting key 2125554136ad860e8ecde91662c95170
Sep 26 14:50:37 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:37.956786 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:37.948300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:41697 (local address 127.5.15.190:38653)
0926 14:50:37.948354 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:37.948356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:37.948545 (+ 189us) server_negotiation.cc:407] Connection header received
0926 14:50:37.949043 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:37.949044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:37.949060 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:37.949110 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:37.949975 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.950494 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.951504 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.951768 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.951826 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:37.954053 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:37.954054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:37.954054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:37.954067 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:37.955437 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:37.955865 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:37.955866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:37.955866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:37.955921 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:37.956171 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:37.956172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:37.956172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:37.956276 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:37.956349 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:37.956615 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:50:37.956700 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:37.956929 13548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:37.948300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:41697)
0926 14:50:37.948435 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:37.948466 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:37.948928 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:37.949177 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:37.949180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:37.949699 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:37.949869 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:37.949873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.950584 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.950586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:37.951407 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:37.951410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:37.951889 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:37.951889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:37.952028 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:37.952472 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:37.952482 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:37.953941 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:37.955521 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:37.955528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:37.955530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:37.955798 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:37.955986 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:37.955987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:37.955987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:37.956096 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:37.956340 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:37.956346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:37.956476 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 14:50:37.956580 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:50:37.956724 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260926 14:50:37.957345 13546 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38653
I20260926 14:50:37.957433 13546 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:37.957597 13546 heartbeater.cc:511] Master 127.5.15.190:38653 requested a full tablet report, sending...
I20260926 14:50:37.958117 13355 ts_manager.cc:194] Registered new tserver with Master: 2fd896d774d846419edabc0214f70943 (127.5.15.129:38681)
I20260926 14:50:37.958706 13355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:41697
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:37.976951 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38653
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.054984 13551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.055203 13551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.055226 13551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.056329 13551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.056363 13551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.056389 13551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:38.057693 13551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:38.057855 13551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.058023 13551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.062640 13559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.062793 13551 server_base.cc:1034] running on GCE node
W20260926 14:50:38.062633 13557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.062633 13556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.063184 13551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.063402 13551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.064536 13551 hybrid_clock.cc:648] HybridClock initialized: now 1790434238064522 us; error 25 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.072973 13551 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:38.073516 13551 webserver.cc:460] Webserver started at http://127.5.15.130:46127/ using document root <none> and password file <none>
I20260926 14:50:38.073736 13551 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.073797 13551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.073925 13551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:38.075066 13551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "65e589fc7414418fb01b2faa0f5f09fd"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "422be6028392df4af46327bdde436bce"
server_key_iv: "71fc67aa6e08ca4e2ba1bd02b54ffde6"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.075408 13551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "65e589fc7414418fb01b2faa0f5f09fd"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "422be6028392df4af46327bdde436bce"
server_key_iv: "71fc67aa6e08ca4e2ba1bd02b54ffde6"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.076633 13551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:38.077313 13566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.077476 13551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:38.077569 13551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "65e589fc7414418fb01b2faa0f5f09fd"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "422be6028392df4af46327bdde436bce"
server_key_iv: "71fc67aa6e08ca4e2ba1bd02b54ffde6"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.077642 13551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.096398 13551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.098678 13551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.098873 13551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.099105 13551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:38.099396 13551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.099447 13551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.099478 13551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.099504 13551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.105268 13551 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:46489
I20260926 14:50:38.105315 13679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:46489 every 8 connection(s)
I20260926 14:50:38.105603 13551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:38.111413 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13551
I20260926 14:50:38.111553 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:38.111748 5182 external_mini_cluster.cc:1428] Setting key 6801cc28a9b8f560de490d97f46941e4
I20260926 14:50:38.115063 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.106282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:49803 (local address 127.5.15.190:38653)
0926 14:50:38.106347 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.106349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.106531 (+ 182us) server_negotiation.cc:407] Connection header received
0926 14:50:38.107071 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.107072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.107084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.107142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.108056 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.108558 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.109544 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.109733 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.109807 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.111904 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.111906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.111906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.111915 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.113639 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.114040 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.114041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.114042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.114108 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.114363 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.114364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.114364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.114483 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.114555 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.114877 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.114974 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:38.115223 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.106261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:49803)
0926 14:50:38.106413 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.106446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.106957 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.107231 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.107234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.107771 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.107945 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.107950 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.108661 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.108662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.109439 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.109441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.109811 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.109811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.109930 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.110343 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.110350 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.111819 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.113720 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.113725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.113727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.113935 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.114188 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.114188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.114189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.114292 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.114555 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.114562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.114699 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.114837 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.114992 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 14:50:38.115648 13680 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38653
I20260926 14:50:38.115747 13680 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:38.115926 13680 heartbeater.cc:511] Master 127.5.15.190:38653 requested a full tablet report, sending...
I20260926 14:50:38.116276 13355 ts_manager.cc:194] Registered new tserver with Master: 65e589fc7414418fb01b2faa0f5f09fd (127.5.15.130:46489)
I20260926 14:50:38.116636 13355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:49803
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:38.134907 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38653
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.214419 13685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.214651 13685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.214689 13685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.215834 13685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.215902 13685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.215946 13685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:38.217381 13685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:38.217648 13685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.217861 13685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.222461 13693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.222455 13691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.222455 13690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.222543 13685 server_base.cc:1034] running on GCE node
I20260926 14:50:38.222854 13685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.223095 13685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.224252 13685 hybrid_clock.cc:648] HybridClock initialized: now 1790434238224228 us; error 39 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.226522 13685 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:38.226981 13685 webserver.cc:460] Webserver started at http://127.5.15.131:43429/ using document root <none> and password file <none>
I20260926 14:50:38.227209 13685 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.227276 13685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.227424 13685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:38.228403 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "44f8da49be1d4e99ab152f926a9e1191"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "3903621bd1ca07d34e7431568c2db29a"
server_key_iv: "85b9b028931180e8ff47ddcfe037dc08"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.228749 13685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "44f8da49be1d4e99ab152f926a9e1191"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "3903621bd1ca07d34e7431568c2db29a"
server_key_iv: "85b9b028931180e8ff47ddcfe037dc08"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.229959 13685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:50:38.230695 13700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.230894 13685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:38.230986 13685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "44f8da49be1d4e99ab152f926a9e1191"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "3903621bd1ca07d34e7431568c2db29a"
server_key_iv: "85b9b028931180e8ff47ddcfe037dc08"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.231076 13685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.241214 13685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.242842 13685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.242987 13685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.243193 13685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:38.243460 13685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.243513 13685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.243548 13685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.243575 13685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.249316 13685 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:34183
I20260926 14:50:38.249349 13813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:34183 every 8 connection(s)
I20260926 14:50:38.249626 13685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:38.257562 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.250255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:59219 (local address 127.5.15.190:38653)
0926 14:50:38.250300 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.250302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.250499 (+ 197us) server_negotiation.cc:407] Connection header received
0926 14:50:38.250977 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.250978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.250988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.251039 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.251847 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.252362 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.253269 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.253441 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.253488 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.255349 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.255350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.255351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.255356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.256498 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.256847 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.256848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.256848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.256895 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.257072 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.257072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.257073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.257167 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.257227 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.257441 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.257499 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:38.257730 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.250253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:59219)
0926 14:50:38.250389 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.250419 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.250896 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.251085 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.251087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.251590 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.251761 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.251765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.252439 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.252440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.253188 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.253190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.253511 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.253512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.253604 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.253972 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.253980 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.255274 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:38.256562 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.256567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.256569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.256784 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.256934 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:38.256934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:38.256934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:38.257016 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:38.257235 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:38.257239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:38.257337 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:50:38.257414 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:50:38.257523 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260926 14:50:38.258262 13814 heartbeater.cc:347] Connected to a master server at 127.5.15.190:38653
I20260926 14:50:38.258360 13814 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:38.258527 13814 heartbeater.cc:511] Master 127.5.15.190:38653 requested a full tablet report, sending...
I20260926 14:50:38.258883 13355 ts_manager.cc:194] Registered new tserver with Master: 44f8da49be1d4e99ab152f926a9e1191 (127.5.15.131:34183)
I20260926 14:50:38.258924 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13685
I20260926 14:50:38.259047 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:38.259285 5182 external_mini_cluster.cc:1428] Setting key 13294831fbe02df9645e1b7ca60798b0
I20260926 14:50:38.259311 13355 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:59219
I20260926 14:50:38.260124 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab.
Entry for principal oryx/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab.
WARNING: no policy specified for oryx/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab.
Entry for principal oryx/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab.
WARNING: no policy specified for oryx/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab.
Entry for principal oryx/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab.
I20260926 14:50:38.318743 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13417
I20260926 14:50:38.323145 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13551
I20260926 14:50:38.327313 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13685
I20260926 14:50:38.332736 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13323
I20260926 14:50:38.337761 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38653
--webserver_interface=127.5.15.190
--webserver_port=39195
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.414441 13823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.414619 13823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.414639 13823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.415791 13823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:38.415831 13823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.415845 13823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.415858 13823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:38.415869 13823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:38.417227 13823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=39195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:38.417403 13823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.417555 13823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.422065 13828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.422230 13829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.422251 13831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.422266 13823 server_base.cc:1034] running on GCE node
I20260926 14:50:38.422592 13823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.422885 13823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.424038 13823 hybrid_clock.cc:648] HybridClock initialized: now 1790434238424031 us; error 37 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.426362 13823 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:38.426878 13823 webserver.cc:460] Webserver started at http://127.5.15.190:39195/ using document root <none> and password file <none>
I20260926 14:50:38.427119 13823 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.427202 13823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.428453 13823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:38.429092 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.429255 13823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:38.429361 13823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "320a17ba06f1405896ca625f08eb8290"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "6ede322046ee24fd1c1d6a2ff344da5d"
server_key_iv: "89fe97753c7cffcdd3ef04bab440e7e4"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.429684 13823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.444808 13823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.447079 13823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.447232 13823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.451114 13823 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:38653
I20260926 14:50:38.451164 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:38653 every 8 connection(s)
I20260926 14:50:38.451534 13823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:38.452224 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13823
I20260926 14:50:38.452528 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:38681
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:38653
--webserver_port=33465
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:50:38.453442 13891 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:50:38.454284 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Bootstrap starting.
I20260926 14:50:38.462630 13891 log.cc:824] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:38.464264 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:50:38.464576 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Bootstrap complete.
I20260926 14:50:38.466848 13891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:38.467094 13891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320a17ba06f1405896ca625f08eb8290, State: Initialized, Role: FOLLOWER
I20260926 14:50:38.467267 13891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:38.467324 13891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:38.467370 13891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:38.467417 13891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:50:38.468377 13891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:38.468504 13891 leader_election.cc:302] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320a17ba06f1405896ca625f08eb8290; no voters:
I20260926 14:50:38.468730 13891 leader_election.cc:288] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:50:38.468762 13895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:50:38.468869 13895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [term 2 LEADER]: Becoming Leader. State: Replica: 320a17ba06f1405896ca625f08eb8290, State: Running, Role: LEADER
I20260926 14:50:38.468984 13891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:38.469005 13895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } }
I20260926 14:50:38.469288 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "320a17ba06f1405896ca625f08eb8290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } } }
I20260926 14:50:38.469286 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320a17ba06f1405896ca625f08eb8290. Latest consensus state: current_term: 2 leader_uuid: "320a17ba06f1405896ca625f08eb8290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a17ba06f1405896ca625f08eb8290" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38653 } } }
I20260926 14:50:38.469461 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:38.469439 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290 [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:38.469722 13904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:38.470199 13904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:38.470340 13904 catalog_manager.cc:1259] Loaded cluster ID: 67a9947448584626836adadafdca637c
I20260926 14:50:38.470376 13904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:38.471508 13904 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:38.471690 13904 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 320a17ba06f1405896ca625f08eb8290: Loaded TSK: 0
I20260926 14:50:38.471894 13904 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:50:38.532459 13893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.532660 13893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.532696 13893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.533962 13893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.534024 13893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.534073 13893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:38.535490 13893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.129.keytab
--principal=oryx/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=33465
--tserver_master_addrs=127.5.15.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:38.535748 13893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.535964 13893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.540910 13920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.541018 13917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.540910 13918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.541260 13893 server_base.cc:1034] running on GCE node
I20260926 14:50:38.541440 13893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.541692 13893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.542802 13893 hybrid_clock.cc:648] HybridClock initialized: now 1790434238542790 us; error 32 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.545291 13893 init.cc:377] Logged in from keytab as oryx/127.5.15.129@KRBTEST.COM (short username oryx)
I20260926 14:50:38.545713 13893 webserver.cc:460] Webserver started at http://127.5.15.129:33465/ using document root <none> and password file <none>
I20260926 14:50:38.545957 13893 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.546041 13893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.547479 13893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:50:38.548095 13927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.548264 13893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.548359 13893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "2fd896d774d846419edabc0214f70943"
format_stamp: "Formatted at 2026-09-26 14:50:37 on dist-test-slave-s4p1"
server_key: "0b0f7f6b1c87ac24a4e7c33c48e37b5a"
server_key_iv: "c14b5663fa94570f50c6e5fdd391a9f1"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.548671 13893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.558400 13893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.560621 13893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.560779 13893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.560985 13893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:38.561262 13893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.561313 13893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.561353 13893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.561380 13893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.567507 13893 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:38681
I20260926 14:50:38.567549 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:38681 every 8 connection(s)
I20260926 14:50:38.567900 13893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.577240 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.568614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:58069)
0926 14:50:38.568800 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.568844 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.569519 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.570170 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.570174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.570824 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.571007 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.571011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.572598 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.572599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.573346 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.573348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.574146 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.574146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.574242 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.574800 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.574817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.577056 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260926 14:50:38.577339 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 13893
W20260926 14:50:38.577442 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.568735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:58069 (local address 127.5.15.190:38653)
0926 14:50:38.569765 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.569773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.569807 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:38.569929 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.569935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.569986 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.570110 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.571127 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.572496 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.573442 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.574052 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.574160 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.577248 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:58069: BlockingRecv error: recv got EOF from 127.5.15.129:58069 (error 108)
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":67,"threads_started":1}
I20260926 14:50:38.577636 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:46489
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:38653
--webserver_port=46127
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:38.577800 14041 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:38653 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.15.190:38653: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.582998 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.578061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:58881)
0926 14:50:38.578139 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.578149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.578183 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.578475 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.578476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.578632 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.578685 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.578687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.579567 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.579569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.580320 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.580322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.580692 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.580693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.580758 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.581185 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.581190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.582910 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 14:50:38.583045 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.578283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:58881 (local address 127.5.15.190:38653)
0926 14:50:38.578347 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.578349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.578358 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:38.578378 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.578379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.578390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.578458 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.578784 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.579469 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.580402 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.580588 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.580678 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.582980 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:58881: BlockingRecv error: recv got EOF from 127.5.15.129:58881 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.587387 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.583482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:54533)
0926 14:50:38.583519 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.583529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.583563 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.583684 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.583685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.583809 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.583832 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.583833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.584431 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.584432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.584913 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.584914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.585283 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.585284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.585335 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.585745 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.585750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.587313 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 14:50:38.587435 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.583531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:54533 (local address 127.5.15.190:38653)
0926 14:50:38.583563 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.583565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.583573 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:38.583616 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.583617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.583626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.583681 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.583906 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.584346 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.585012 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.585202 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.585283 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.587374 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:54533: BlockingRecv error: recv got EOF from 127.5.15.129:54533 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 14:50:38.587729 14041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:50:38.662847 14045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.663034 14045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.663056 14045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.664129 14045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.664163 14045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.664187 14045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:38.665493 14045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.130.keytab
--principal=oryx/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:46489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=46127
--tserver_master_addrs=127.5.15.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:38.665668 14045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.665825 14045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.670583 14050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.670825 14045 server_base.cc:1034] running on GCE node
W20260926 14:50:38.670591 14051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.670668 14053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.671166 14045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.671417 14045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.672581 14045 hybrid_clock.cc:648] HybridClock initialized: now 1790434238672566 us; error 27 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.674939 14045 init.cc:377] Logged in from keytab as oryx/127.5.15.130@KRBTEST.COM (short username oryx)
I20260926 14:50:38.675343 14045 webserver.cc:460] Webserver started at http://127.5.15.130:46127/ using document root <none> and password file <none>
I20260926 14:50:38.675581 14045 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.675662 14045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.676909 14045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.677503 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.677673 14045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:38.677767 14045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "65e589fc7414418fb01b2faa0f5f09fd"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "422be6028392df4af46327bdde436bce"
server_key_iv: "71fc67aa6e08ca4e2ba1bd02b54ffde6"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.678107 14045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.694083 14045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.695884 14045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.696043 14045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.696252 14045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:38.696539 14045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.696596 14045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.696627 14045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.696655 14045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.702404 14045 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:46489
I20260926 14:50:38.702450 14173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:46489 every 8 connection(s)
I20260926 14:50:38.702762 14045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:38.703135 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14045
I20260926 14:50:38.703430 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:34183
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:38653
--webserver_port=43429
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:38.710235 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.703465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:32969 (local address 127.5.15.190:38653)
0926 14:50:38.703532 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.703534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.703754 (+ 220us) server_negotiation.cc:407] Connection header received
0926 14:50:38.704302 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.704303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.704321 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.704392 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.705293 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.705893 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.706958 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.707159 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.707261 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.710169 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:32969: BlockingRecv error: recv got EOF from 127.5.15.130:32969 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:50:38.710289 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.703447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:32969)
0926 14:50:38.703620 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.703653 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.704186 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.704443 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.704446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.704951 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.705180 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.705186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.706000 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.706001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.706873 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.706875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.707271 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.707272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.707403 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.707920 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.707928 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.710096 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
W20260926 14:50:38.710530 14174 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:38653 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.15.190:38653: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.714911 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.710801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:39755)
0926 14:50:38.710875 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.710884 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.710927 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.711046 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.711046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.711176 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.711216 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.711217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.711805 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.711806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.712293 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.712294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.712637 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.712638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.712699 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.713109 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.713112 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.714851 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 14:50:38.715013 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.710831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:39755 (local address 127.5.15.190:38653)
0926 14:50:38.710867 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.710869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.710932 (+ 63us) server_negotiation.cc:407] Connection header received
0926 14:50:38.710983 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.710984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.710993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.711042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.711295 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.711727 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.712387 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.712558 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.712636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.714928 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:39755: BlockingRecv error: recv got EOF from 127.5.15.130:39755 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.719133 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.715139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:49173)
0926 14:50:38.715204 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.715214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.715259 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.715395 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.715395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.715531 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.715555 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.715556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.716140 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.716141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.716616 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.716618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.716946 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.716947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.717008 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.717414 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.717417 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.719066 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 14:50:38.719183 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.715169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:49173 (local address 127.5.15.190:38653)
0926 14:50:38.715197 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.715199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.715246 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:50:38.715315 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.715315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.715324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.715391 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.715641 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.716067 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.716696 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.716877 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.716948 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.719122 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:49173: BlockingRecv error: recv got EOF from 127.5.15.130:49173 (error 108)
Metrics: {"server-negotiator.queue_time_us":12}
W20260926 14:50:38.719534 14174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:50:38.783375 14177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:38.783555 14177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:38.783577 14177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:38.784602 14177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:38.784633 14177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:38.784657 14177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:38.785908 14177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/krb5kdc/oryx_127.5.15.131.keytab
--principal=oryx/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:34183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=43429
--tserver_master_addrs=127.5.15.190:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:38.786059 14177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:38.786199 14177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:38.790683 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.790865 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:38.790777 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:38.790767 14177 server_base.cc:1034] running on GCE node
I20260926 14:50:38.791157 14177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:38.791420 14177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:38.792577 14177 hybrid_clock.cc:648] HybridClock initialized: now 1790434238792562 us; error 32 us; skew 500 ppm
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434238, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:38.794891 14177 init.cc:377] Logged in from keytab as oryx/127.5.15.131@KRBTEST.COM (short username oryx)
I20260926 14:50:38.795286 14177 webserver.cc:460] Webserver started at http://127.5.15.131:43429/ using document root <none> and password file <none>
I20260926 14:50:38.795507 14177 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:38.795604 14177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:38.796808 14177 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:38.797374 14192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.797539 14177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:38.797648 14177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "44f8da49be1d4e99ab152f926a9e1191"
format_stamp: "Formatted at 2026-09-26 14:50:38 on dist-test-slave-s4p1"
server_key: "3903621bd1ca07d34e7431568c2db29a"
server_key_iv: "85b9b028931180e8ff47ddcfe037dc08"
server_key_version: "encryptionkey@0"
I20260926 14:50:38.797963 14177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:38.820787 14177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:38.822405 14177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:38.822556 14177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:38.822811 14177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:38.823112 14177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:38.823175 14177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.823208 14177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:38.823233 14177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:38.828648 14177 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:34183
I20260926 14:50:38.828717 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:34183 every 8 connection(s)
I20260926 14:50:38.829038 14177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:50:38.835664 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.829714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:49001 (local address 127.5.15.190:38653)
0926 14:50:38.829784 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.829787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.829985 (+ 198us) server_negotiation.cc:407] Connection header received
0926 14:50:38.830377 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.830378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.830387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.830432 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.831312 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.831768 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.832770 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.832941 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.832992 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.835612 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:49001: BlockingRecv error: recv got EOF from 127.5.15.131:49001 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:50:38.835704 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.829710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:49001)
0926 14:50:38.829869 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.829899 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.830300 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.830491 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.830493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.831075 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.831242 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.831246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.831849 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.831850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.832703 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.832705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.833010 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.833010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.833101 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.833560 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.833568 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.835531 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260926 14:50:38.835880 14306 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:38653 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.15.190:38653: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
I20260926 14:50:38.838176 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14177
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.840135 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.836096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:45929)
0926 14:50:38.836140 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.836150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.836182 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.836416 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.836417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.836519 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.836558 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.836559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.837070 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.837070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.837543 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.837544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.837818 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.837819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.837859 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.838263 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.838267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.840071 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:50:38.840183 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.836266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:45929 (local address 127.5.15.190:38653)
0926 14:50:38.836315 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.836317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.836324 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:38.836353 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.836353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.836359 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.836403 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.836614 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.837013 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.837604 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.837762 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.837806 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.840128 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:45929: BlockingRecv error: recv got EOF from 127.5.15.131:45929 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:38 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:38.844875 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.840399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:48095)
0926 14:50:38.840479 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:38.840488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:38.840532 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:38.840622 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:38.840622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:38.840740 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:38.840764 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.840765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.841616 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.841617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.842394 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:38.842396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.842810 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.842811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:38.842865 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.843327 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:38.843330 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:38.844832 (+ 1502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 14:50:38.844970 14044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:38.840439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:48095 (local address 127.5.15.190:38653)
0926 14:50:38.840472 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.840474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.840521 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:50:38.840571 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.840572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.840577 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.840611 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.840896 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.841547 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.842483 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.842711 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.842810 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.844910 (+ 2100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:48095: BlockingRecv error: recv got EOF from 127.5.15.131:48095 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 14:50:38.845110 14306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:50:38.845683 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:38.838586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.5.15.190:38653)
0926 14:50:38.838689 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:38.838690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:38.838795 (+ 105us) server_negotiation.cc:407] Connection header received
0926 14:50:38.838864 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:38.838865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:38.838877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:38.838953 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:38.839541 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.840040 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.841029 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:38.841332 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:38.841437 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:38.842616 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:38.842620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:38.842620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:38.842630 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:38.844369 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.844757 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.844760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.844761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.844888 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:38.845111 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:38.845112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:38.845112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:38.845357 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:38.845458 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:38.845555 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 14:50:38.845631 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
Sep 26 14:50:39 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:39 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:39.592538 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.588129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:50771)
0926 14:50:39.588269 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.588279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.588331 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.588466 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.588467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.588617 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:39.588663 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.588664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.589274 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.589275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.589806 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.589807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.590124 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.590125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.590164 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.590605 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:39.590609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:39.592418 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
W20260926 14:50:39.592577 14312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:39.588207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:50771 (local address 127.5.15.190:38653)
0926 14:50:39.588315 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.588317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.588327 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:39.588377 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.588378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.588387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.588445 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:39.588729 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.589212 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.589877 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.590063 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.590110 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.592496 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:50771: BlockingRecv error: recv got EOF from 127.5.15.129:50771 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
Sep 26 14:50:39 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:39 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:39.724833 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.720135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:40381)
0926 14:50:39.720321 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.720351 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.720423 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.720596 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.720597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.720779 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:39.720824 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.720826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.721486 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.721487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.722032 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.722034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.722412 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.722413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.722464 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.723007 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:39.723012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:39.724729 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
W20260926 14:50:39.724869 14312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:39.720235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:40381 (local address 127.5.15.190:38653)
0926 14:50:39.720322 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.720324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.720403 (+ 79us) server_negotiation.cc:407] Connection header received
0926 14:50:39.720478 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.720479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.720492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.720586 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:39.720906 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.721400 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.722127 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.722335 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.722413 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.724805 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:40381: BlockingRecv error: recv got EOF from 127.5.15.130:40381 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 14:50:39 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:39 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:39.849992 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:39.845432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:54199)
0926 14:50:39.845569 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:39.845581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:39.845660 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:39.845803 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:39.845804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:39.845966 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:39.846008 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.846010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.846609 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.846610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.847228 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:39.847230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.847637 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.847638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:39.847694 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.848185 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:39.848190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:39.849890 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260926 14:50:39.850033 14312 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:39.845507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:54199 (local address 127.5.15.190:38653)
0926 14:50:39.845572 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:39.845574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:39.845643 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:50:39.845720 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:39.845721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:39.845734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:39.845796 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:39.846078 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.846527 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.847308 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:39.847552 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:39.847637 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:39.849968 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:54199: BlockingRecv error: recv got EOF from 127.5.15.131:54199 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:50:40 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:40 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:40.597895 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:40.593126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:52735)
0926 14:50:40.593309 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:40.593321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:40.593366 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:40.593544 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:40.593545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:40.593743 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:40.593801 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:40.593804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.594449 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.594450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:40.594993 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:40.594994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.595445 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.595446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:40.595506 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:40.596014 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:40.596020 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:40.597776 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":120,"threads_started":1}
W20260926 14:50:40.597944 14316 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:40.593244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:52735 (local address 127.5.15.190:38653)
0926 14:50:40.593360 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:40.593362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:40.593402 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:50:40.593424 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:40.593425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:40.593436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:40.593508 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:40.593869 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.594355 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.595117 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.595319 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.595431 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:40.597873 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:52735: BlockingRecv error: recv got EOF from 127.5.15.129:52735 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
Sep 26 14:50:40 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:40 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:40.729885 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:40.725430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:55559)
0926 14:50:40.725598 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:40.725610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:40.725690 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:40.725811 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:40.725811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:40.725996 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:40.726042 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:40.726044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.726687 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.726688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:40.727234 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:40.727236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.727616 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.727617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:40.727674 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:40.728117 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:40.728121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:40.729823 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
W20260926 14:50:40.730007 14316 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:40.725513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:55559 (local address 127.5.15.190:38653)
0926 14:50:40.725582 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:40.725584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:40.725674 (+ 90us) server_negotiation.cc:407] Connection header received
0926 14:50:40.725744 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:40.725745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:40.725758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:40.725806 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:40.726118 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.726603 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.727318 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.727532 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.727616 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:40.729910 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:55559: BlockingRecv error: recv got EOF from 127.5.15.130:55559 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 14:50:40 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:40 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:40.855376 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:40.850632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:51795)
0926 14:50:40.850797 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:40.850810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:40.850874 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:40.851053 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:40.851053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:40.851220 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:40.851264 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:40.851266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.852127 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.852128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:40.852632 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:40.852634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.852997 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.852998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:40.853054 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:40.853482 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:40.853486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:40.855304 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":54,"threads_started":1}
W20260926 14:50:40.855472 14316 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:40.850765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:51795 (local address 127.5.15.190:38653)
0926 14:50:40.850804 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:40.850806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:40.850855 (+ 49us) server_negotiation.cc:407] Connection header received
0926 14:50:40.850956 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:40.850957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:40.850971 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:40.851046 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:40.851347 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.852057 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.852725 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:40.852923 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:40.852998 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:40.855400 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:51795: BlockingRecv error: recv got EOF from 127.5.15.131:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 14:50:41 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:41 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:41.602705 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:41.598369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:60979)
0926 14:50:41.598520 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:41.598530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:41.598575 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:41.598688 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:41.598689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:41.598869 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:41.598916 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:41.598918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.599532 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.599533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:41.600050 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:41.600051 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.600373 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.600374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:41.600409 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:41.600879 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:41.600883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:41.602640 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
W20260926 14:50:41.602857 14320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:41.598461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:60979 (local address 127.5.15.190:38653)
0926 14:50:41.598555 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:41.598557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:41.598585 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:50:41.598625 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:41.598626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:41.598636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:41.598685 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:41.598974 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.599466 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.600121 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.600313 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.600362 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:41.602728 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:60979: BlockingRecv error: recv got EOF from 127.5.15.129:60979 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":58,"threads_started":1}
Sep 26 14:50:41 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:41 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:41.734709 14321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:41.730364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:39797)
0926 14:50:41.730487 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:41.730498 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:41.730569 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:41.730692 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:41.730693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:41.730901 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:41.730947 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:41.730949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.731546 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.731546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:41.732051 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:41.732052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.732347 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.732348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:41.732391 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:41.732911 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:41.732916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:41.734654 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
W20260926 14:50:41.734831 14320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:41.730451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:39797 (local address 127.5.15.190:38653)
0926 14:50:41.730497 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:41.730499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:41.730558 (+ 59us) server_negotiation.cc:407] Connection header received
0926 14:50:41.730624 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:41.730624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:41.730634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:41.730674 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:41.731009 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.731486 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.732111 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.732289 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.732333 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:41.734733 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:39797: BlockingRecv error: recv got EOF from 127.5.15.130:39797 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 14:50:41 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:41 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:41.860404 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:41.855826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:38825)
0926 14:50:41.855987 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:41.856001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:41.856095 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:41.856238 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:41.856239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:41.856421 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:41.856466 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:41.856468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.857118 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.857119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:41.857674 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:41.857675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.858039 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.858040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:41.858093 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:41.858532 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:41.858536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:41.860335 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260926 14:50:41.860532 14320 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:41.855911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38825 (local address 127.5.15.190:38653)
0926 14:50:41.855985 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:41.855986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:41.856076 (+ 90us) server_negotiation.cc:407] Connection header received
0926 14:50:41.856151 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:41.856151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:41.856170 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:41.856232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:41.856560 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.857035 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.857765 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:41.857966 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:41.858043 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:41.860448 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:38825: BlockingRecv error: recv got EOF from 127.5.15.131:38825 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:50:42 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:42 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:42.607677 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.603236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:49311)
0926 14:50:42.603395 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:42.603407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:42.603472 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:42.603689 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:42.603690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:42.603938 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:42.603992 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.603995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.604629 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.604630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.605146 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.605148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.605452 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.605453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.605490 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.605938 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:42.605942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:42.607620 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
W20260926 14:50:42.607784 14324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:42.603305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:49311 (local address 127.5.15.190:38653)
0926 14:50:42.603490 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:42.603492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:42.603526 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:42.603549 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:42.603550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:42.603562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:42.603669 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:42.604063 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.604562 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.605213 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.605394 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.605441 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.607710 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:49311: BlockingRecv error: recv got EOF from 127.5.15.129:49311 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
Sep 26 14:50:42 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:42 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:42.739485 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.735184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:57259)
0926 14:50:42.735330 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:42.735341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:42.735404 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:42.735522 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:42.735522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:42.735676 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:42.735722 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.735723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.736332 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.736333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.736838 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.736839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.737145 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.737145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.737182 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.737631 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:42.737636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:42.739391 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260926 14:50:42.739530 14324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:42.735256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:57259 (local address 127.5.15.190:38653)
0926 14:50:42.735340 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:42.735342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:42.735390 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:50:42.735453 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:42.735453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:42.735465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:42.735518 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:42.735782 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.736267 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.736904 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.737086 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.737135 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.739469 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:57259: BlockingRecv error: recv got EOF from 127.5.15.130:57259 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 14:50:42 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:42 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:42.865612 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:42.861059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:46269)
0926 14:50:42.861228 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:42.861239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:42.861294 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:42.861427 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:42.861428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:42.861585 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:42.861632 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.861634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.862243 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.862244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.862807 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:42.862809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.863188 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.863189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:42.863248 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.863773 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:42.863778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:42.865509 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
W20260926 14:50:42.865671 14324 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:42.861151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:46269 (local address 127.5.15.190:38653)
0926 14:50:42.861208 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:42.861209 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:42.861298 (+ 89us) server_negotiation.cc:407] Connection header received
0926 14:50:42.861338 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:42.861338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:42.861349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:42.861407 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:42.861693 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.862180 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.862921 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:42.863103 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:42.863185 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:42.865611 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:46269: BlockingRecv error: recv got EOF from 127.5.15.131:46269 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 14:50:43 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.612517 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.608139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:54855)
0926 14:50:43.608305 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.608315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.608369 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.608513 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.608514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.608667 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.608714 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.608715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.609326 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.609327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.609838 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.609839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.610146 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.610146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.610187 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.610635 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.610640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.612448 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
W20260926 14:50:43.612638 14328 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.608226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:54855 (local address 127.5.15.190:38653)
0926 14:50:43.608333 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.608335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.608360 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:50:43.608429 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.608429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.608439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.608497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.608774 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.609266 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.609910 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.610084 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.610132 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.612534 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:54855: BlockingRecv error: recv got EOF from 127.5.15.129:54855 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
Sep 26 14:50:43 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.744535 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.740122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:38327)
0926 14:50:43.740273 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.740283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.740334 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.740461 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.740462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.740616 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.740661 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.740663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.741260 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.741261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.741742 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.741743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.742042 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.742042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.742078 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.742610 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.742615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.744432 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260926 14:50:43.744570 14328 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.740263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:38327 (local address 127.5.15.190:38653)
0926 14:50:43.740309 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.740311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.740323 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:43.740387 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.740387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.740396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.740442 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.740724 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.741198 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.741806 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.741984 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.742030 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.744506 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:38327: BlockingRecv error: recv got EOF from 127.5.15.130:38327 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 14:50:43 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:43 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:43.871137 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:43.866248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:58319)
0926 14:50:43.866433 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:43.866445 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:43.866544 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:43.866710 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:43.866710 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:43.866901 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:43.866964 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.866966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.867670 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.867671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.868198 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:43.868200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.868630 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.868631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:43.868697 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.869117 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:43.869122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:43.871044 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
W20260926 14:50:43.871189 14328 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:43.866343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:58319 (local address 127.5.15.190:38653)
0926 14:50:43.866429 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:43.866431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:43.866527 (+ 96us) server_negotiation.cc:407] Connection header received
0926 14:50:43.866619 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:43.866619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:43.866634 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:43.866704 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:43.867044 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.867571 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.868282 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:43.868543 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:43.868628 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:43.871121 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:58319: BlockingRecv error: recv got EOF from 127.5.15.131:58319 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 14:50:44 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:44 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:44.617708 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:44.613143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:33087)
0926 14:50:44.613298 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:44.613308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:44.613361 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:44.613499 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:44.613500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:44.613650 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:44.613698 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.613700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.614288 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.614288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.614836 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.614837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.615137 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.615137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.615175 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.615722 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:44.615726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:44.617619 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
W20260926 14:50:44.617758 14332 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:44.613241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:33087 (local address 127.5.15.190:38653)
0926 14:50:44.613376 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:44.613377 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:44.613386 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:44.613414 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:44.613414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:44.613424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:44.613482 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:44.613761 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.614227 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.614898 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.615075 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.615124 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.617691 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:33087: BlockingRecv error: recv got EOF from 127.5.15.129:33087 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 26 14:50:44 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:44 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:44.749390 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:44.744981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:38751)
0926 14:50:44.745132 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:44.745144 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:44.745204 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:44.745385 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:44.745386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:44.745590 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:44.745644 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.745646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.746223 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.746223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.746705 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.746706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.747081 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.747082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.747120 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.747577 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:44.747581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:44.749299 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260926 14:50:44.749437 14332 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:44.745141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:38751 (local address 127.5.15.190:38653)
0926 14:50:44.745186 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:44.745188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:44.745215 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:50:44.745264 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:44.745265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:44.745278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:44.745364 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:44.745713 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.746161 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.746845 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.747022 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.747073 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.749374 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:38751: BlockingRecv error: recv got EOF from 127.5.15.130:38751 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 14:50:44 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:44 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:44.876006 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:44.871693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:53733)
0926 14:50:44.871826 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:44.871844 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:44.871910 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:44.872034 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:44.872034 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:44.872192 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:44.872248 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.872250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.872840 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.872841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.873338 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:44.873339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.873629 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.873629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:44.873665 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.874128 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:44.874132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:44.875952 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
W20260926 14:50:44.876147 14332 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:44.871753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:53733 (local address 127.5.15.190:38653)
0926 14:50:44.871822 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:44.871824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:44.871905 (+ 81us) server_negotiation.cc:407] Connection header received
0926 14:50:44.871953 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:44.871954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:44.871964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:44.872017 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:44.872303 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.872774 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.873401 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:44.873572 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:44.873618 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:44.876031 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:53733: BlockingRecv error: recv got EOF from 127.5.15.131:53733 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 14:50:45 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:45 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:45.622596 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:45.618160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:51743)
0926 14:50:45.618311 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:45.618322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:45.618379 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:45.618505 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:45.618506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:45.618661 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:45.618711 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.618713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.619443 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.619444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.619976 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.619978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.620304 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.620304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.620347 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.620832 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:45.620837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:45.622539 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
W20260926 14:50:45.622727 14336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:45.618236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:51743 (local address 127.5.15.190:38653)
0926 14:50:45.618367 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:45.618369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:45.618389 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:50:45.618419 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:45.618419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:45.618428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:45.618486 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:45.618847 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.619335 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.620047 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.620235 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.620286 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.622624 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:51743: BlockingRecv error: recv got EOF from 127.5.15.129:51743 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
Sep 26 14:50:45 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:45 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:45.754415 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:45.749892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:58871)
0926 14:50:45.750072 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:45.750082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:45.750163 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:45.750291 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:45.750292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:45.750491 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:45.750552 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.750554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.751278 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.751280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.751856 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.751858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.752225 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.752226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.752281 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.752695 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:45.752700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:45.754349 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
W20260926 14:50:45.754535 14336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:45.749987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:58871 (local address 127.5.15.190:38653)
0926 14:50:45.750056 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:45.750058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:45.750145 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:50:45.750225 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:45.750226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:45.750238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:45.750287 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:45.750630 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.751193 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.751942 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.752142 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.752223 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.754439 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:58871: BlockingRecv error: recv got EOF from 127.5.15.130:58871 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 14:50:45 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:45 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:45.880676 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:45.876422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:37231)
0926 14:50:45.876552 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:45.876569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:45.876624 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:45.876748 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:45.876749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:45.876901 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:45.876948 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.876950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.877551 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.877551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.878073 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:45.878075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.878374 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.878375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:45.878413 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.878945 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:45.878949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:45.880618 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260926 14:50:45.880844 14336 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:45.876484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:37231 (local address 127.5.15.190:38653)
0926 14:50:45.876531 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:45.876532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:45.876626 (+ 94us) server_negotiation.cc:407] Connection header received
0926 14:50:45.876676 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:45.876677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:45.876687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:45.876728 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:45.877010 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.877488 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.878142 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:45.878315 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:45.878362 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:45.880704 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:37231: BlockingRecv error: recv got EOF from 127.5.15.131:37231 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 14:50:46 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:46 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:46.627671 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:46.623038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:41959)
0926 14:50:46.623186 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:46.623196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:46.623243 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:46.623400 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:46.623401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:46.623558 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:46.623605 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.623606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.624249 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.624249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.624823 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.624826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.625363 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.625364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.625433 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.625886 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:46.625890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:46.627614 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
W20260926 14:50:46.627789 14340 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:46.623159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:41959 (local address 127.5.15.190:38653)
0926 14:50:46.623286 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:46.623287 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:46.623295 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:46.623315 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:46.623316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:46.623324 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:46.623381 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:46.623670 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.624169 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.624917 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.625278 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.625349 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.627706 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:41959: BlockingRecv error: recv got EOF from 127.5.15.129:41959 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
Sep 26 14:50:46 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:46 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:46.759285 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:46.754842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:35003)
0926 14:50:46.754958 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:46.754972 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:46.755030 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:46.755144 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:46.755145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:46.755292 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:46.755337 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.755339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.756078 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.756079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.756569 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.756571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.756882 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.756883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.756937 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.757406 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:46.757410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:46.759182 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
W20260926 14:50:46.759320 14340 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:46.754895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:35003 (local address 127.5.15.190:38653)
0926 14:50:46.754973 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:46.754974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:46.755019 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:50:46.755072 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:46.755072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:46.755082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:46.755131 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:46.755410 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.756018 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.756637 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.756804 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.756851 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.759262 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:35003: BlockingRecv error: recv got EOF from 127.5.15.130:35003 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 14:50:46 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:46 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:46.885795 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:46.881081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:46481)
0926 14:50:46.881238 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:46.881247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:46.881301 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:46.881471 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:46.881472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:46.881631 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:46.881697 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.881698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.882521 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.882522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.883098 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:46.883100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.883399 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.883400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:46.883438 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.883935 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:46.883958 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:46.885705 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260926 14:50:46.885834 14340 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:46.881149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:46481 (local address 127.5.15.190:38653)
0926 14:50:46.881204 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:46.881206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:46.881291 (+ 85us) server_negotiation.cc:407] Connection header received
0926 14:50:46.881378 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:46.881378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:46.881391 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:46.881443 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:46.881769 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.882458 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.883167 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:46.883340 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:46.883397 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:46.885776 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:46481: BlockingRecv error: recv got EOF from 127.5.15.131:46481 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 14:50:47 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:47 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:47.632655 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:47.628249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:42757)
0926 14:50:47.628434 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:47.628448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:47.628515 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:47.628664 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:47.628665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:47.628814 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:47.628867 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.628869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.629489 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.629490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.630038 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.630039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.630354 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.630355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.630394 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.630867 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:47.630871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:47.632548 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
W20260926 14:50:47.632746 14344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:47.628346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:42757 (local address 127.5.15.190:38653)
0926 14:50:47.628500 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:47.628502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:47.628512 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:50:47.628572 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:47.628573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:47.628582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:47.628639 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:47.628934 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.629417 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.630109 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.630293 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.630341 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.632674 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:42757: BlockingRecv error: recv got EOF from 127.5.15.129:42757 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
Sep 26 14:50:47 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:47 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:47.764408 14345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:47.759737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:57547)
0926 14:50:47.759874 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:47.759887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:47.759953 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:47.760075 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:47.760076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:47.760233 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:47.760281 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.760282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.760944 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.760944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.761454 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.761456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.761755 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.761755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.761792 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.762295 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:47.762309 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:47.764309 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260926 14:50:47.764444 14344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:47.759793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:57547 (local address 127.5.15.190:38653)
0926 14:50:47.759877 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:47.759879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:47.759936 (+ 57us) server_negotiation.cc:407] Connection header received
0926 14:50:47.759997 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:47.759997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:47.760008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:47.760055 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:47.760346 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.760879 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.761519 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.761695 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.761742 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.764387 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:57547: BlockingRecv error: recv got EOF from 127.5.15.130:57547 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 14:50:47 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:47 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:47.890894 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:47.886293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:33397)
0926 14:50:47.886474 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:47.886486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:47.886576 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:47.886772 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:47.886773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:47.886952 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:47.886995 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.886997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.887653 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.887654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.888206 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:47.888208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.888570 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.888571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:47.888638 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.889081 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:47.889086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:47.890801 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":98,"threads_started":1}
W20260926 14:50:47.890944 14344 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:47.886372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:33397 (local address 127.5.15.190:38653)
0926 14:50:47.886460 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:47.886463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:47.886559 (+ 96us) server_negotiation.cc:407] Connection header received
0926 14:50:47.886659 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:47.886660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:47.886674 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:47.886745 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:47.887074 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.887569 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.888293 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:47.888487 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:47.888560 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:47.890880 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:33397: BlockingRecv error: recv got EOF from 127.5.15.131:33397 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 14:50:48 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:48 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:48.637974 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:48.633094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:33665)
0926 14:50:48.633251 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:48.633265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:48.633333 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:48.633464 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:48.633465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:48.633626 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:48.633675 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.633676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.634686 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.634687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.635286 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.635289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.635696 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.635696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.635756 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.636198 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:48.636202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:48.637910 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
W20260926 14:50:48.638106 14348 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:48.633165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:33665 (local address 127.5.15.190:38653)
0926 14:50:48.633292 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:48.633294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:48.633316 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:50:48.633375 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:48.633375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:48.633385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:48.633440 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:48.633734 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.634598 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.635378 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.635610 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.635693 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.638002 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:33665: BlockingRecv error: recv got EOF from 127.5.15.129:33665 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
Sep 26 14:50:48 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:48 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:48.769563 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:48.765011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:54531)
0926 14:50:48.765158 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:48.765169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:48.765232 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:48.765354 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:48.765354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:48.765531 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:48.765578 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.765580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.766165 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.766166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.766664 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.766665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.767038 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.767039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.767113 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.767626 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:48.767632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:48.769469 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
W20260926 14:50:48.769631 14348 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:48.765056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:54531 (local address 127.5.15.190:38653)
0926 14:50:48.765128 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:48.765129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:48.765235 (+ 106us) server_negotiation.cc:407] Connection header received
0926 14:50:48.765278 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:48.765279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:48.765289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:48.765338 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:48.765639 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.766104 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.766729 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.766969 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.767037 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.769563 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:54531: BlockingRecv error: recv got EOF from 127.5.15.130:54531 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 14:50:48 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:48 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:48.895310 14350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:48.891309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:37753)
0926 14:50:48.891426 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:48.891436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:48.891487 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:48.891591 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:48.891591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:48.891737 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:48.891783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.891784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.892375 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.892376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.892850 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:48.892851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.893126 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.893126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:48.893162 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.893550 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:48.893553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:48.895247 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
W20260926 14:50:48.895385 14348 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:48.891425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:37753 (local address 127.5.15.190:38653)
0926 14:50:48.891488 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:48.891489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:48.891497 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:48.891520 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:48.891521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:48.891530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:48.891585 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:48.891841 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.892306 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.892912 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:48.893071 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:48.893114 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:48.895337 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:37753: BlockingRecv error: recv got EOF from 127.5.15.131:37753 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 14:50:49 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:49 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:49.642720 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:49.638412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:55091)
0926 14:50:49.638563 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:49.638573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:49.638631 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:49.638770 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:49.638771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:49.638935 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:49.638981 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.638982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.639592 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.639593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.640098 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.640100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.640413 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.640414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.640477 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.640917 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:49.640921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:49.642653 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
W20260926 14:50:49.642892 14352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:49.638483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:55091 (local address 127.5.15.190:38653)
0926 14:50:49.638586 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:49.638587 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:49.638638 (+ 51us) server_negotiation.cc:407] Connection header received
0926 14:50:49.638675 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:49.638675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:49.638686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:49.638783 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:49.639042 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.639526 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.640165 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.640354 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.640401 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.642738 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:55091: BlockingRecv error: recv got EOF from 127.5.15.129:55091 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":51,"threads_started":1}
Sep 26 14:50:49 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:49 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:49.774523 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:49.770015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:59625)
0926 14:50:49.770169 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:49.770180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:49.770260 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:49.770420 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:49.770421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:49.770613 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:49.770656 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.770658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.771375 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.771376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.771901 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.771902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.772279 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.772280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.772348 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.772800 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:49.772804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:49.774421 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
W20260926 14:50:49.774561 14352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:49.770078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:59625 (local address 127.5.15.190:38653)
0926 14:50:49.770151 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:49.770153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:49.770242 (+ 89us) server_negotiation.cc:407] Connection header received
0926 14:50:49.770329 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:49.770330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:49.770342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:49.770412 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:49.770735 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.771283 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.771994 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.772202 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.772270 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.774500 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:59625: BlockingRecv error: recv got EOF from 127.5.15.130:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 14:50:49 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:49 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:49.899878 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:49.895740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:38379)
0926 14:50:49.895866 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:49.895877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:49.895955 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:49.896078 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:49.896079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:49.896241 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:49.896284 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.896285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.896865 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.896866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.897360 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:49.897361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.897647 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.897648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:49.897686 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.898143 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:49.898147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:49.899812 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
W20260926 14:50:49.899971 14352 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:49.895803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38379 (local address 127.5.15.190:38653)
0926 14:50:49.895880 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:49.895881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:49.895955 (+ 74us) server_negotiation.cc:407] Connection header received
0926 14:50:49.896006 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:49.896006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:49.896017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:49.896058 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:49.896341 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.896804 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.897424 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:49.897590 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:49.897636 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:49.899904 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:38379: BlockingRecv error: recv got EOF from 127.5.15.131:38379 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 14:50:50 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:50 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:50.647485 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:50.643164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:37393)
0926 14:50:50.643306 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:50.643319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:50.643388 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:50.643541 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:50.643542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:50.643701 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:50.643746 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.643747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.644394 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.644395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.644918 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.644919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.645225 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.645226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.645263 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.645737 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:50.645741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:50.647420 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260926 14:50:50.647588 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:50.643240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:37393 (local address 127.5.15.190:38653)
0926 14:50:50.643369 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:50.643371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:50.643379 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:50.643449 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:50.643449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:50.643459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:50.643516 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:50.643834 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.644333 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.644985 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.645168 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.645216 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.647513 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:37393: BlockingRecv error: recv got EOF from 127.5.15.129:37393 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
Sep 26 14:50:50 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:50 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:50.779245 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:50.774989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:49465)
0926 14:50:50.775098 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:50.775111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:50.775176 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:50.775297 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:50.775297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:50.775465 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:50.775509 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.775511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.776140 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.776141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.776655 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.776657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.776979 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.776980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.777017 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.777470 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:50.777474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:50.779178 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
W20260926 14:50:50.779340 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:50.775056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:49465 (local address 127.5.15.190:38653)
0926 14:50:50.775133 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:50.775135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:50.775171 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:50:50.775221 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:50.775222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:50.775234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:50.775279 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:50.775570 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.776079 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.776725 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.776916 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.776970 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.779272 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:49465: BlockingRecv error: recv got EOF from 127.5.15.130:49465 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 14:50:50 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:50 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:50.904960 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:50.900522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:55605)
0926 14:50:50.900666 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:50.900684 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:50.900757 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:50.900920 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:50.900921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:50.901163 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:50.901216 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.901218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.901895 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.901896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.902398 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:50.902400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.902696 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.902697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:50.902734 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.903172 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:50.903176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:50.904900 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
W20260926 14:50:50.905097 14356 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:50.900605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:55605 (local address 127.5.15.190:38653)
0926 14:50:50.900658 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:50.900660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:50.900726 (+ 66us) server_negotiation.cc:407] Connection header received
0926 14:50:50.900830 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:50.900830 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:50.900845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:50.900899 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:50.901296 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.901830 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.902469 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:50.902638 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:50.902685 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:50.905015 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:55605: BlockingRecv error: recv got EOF from 127.5.15.131:55605 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 14:50:51 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:51 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:51.652328 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:51.647925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:47465)
0926 14:50:51.648076 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:51.648086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:51.648140 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:51.648283 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:51.648283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:51.648443 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:51.648489 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.648490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.649121 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.649121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.649634 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.649636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.649956 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.649957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.649994 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.650411 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:51.650415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:51.652221 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
W20260926 14:50:51.652366 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:51.647999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47465 (local address 127.5.15.190:38653)
0926 14:50:51.648101 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:51.648103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:51.648125 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:50:51.648212 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:51.648213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:51.648223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:51.648279 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:51.648554 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.649055 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.649699 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.649880 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.649944 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.652295 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:47465: BlockingRecv error: recv got EOF from 127.5.15.129:47465 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
Sep 26 14:50:51 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:51 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:51.784539 14361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:51.779719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:38839)
0926 14:50:51.779856 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:51.779867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:51.779924 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:51.780063 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:51.780063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:51.780250 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:51.780326 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.780328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.780996 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.780998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.781566 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.781568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.781963 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.781964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.782038 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.782580 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:51.782585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:51.784436 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
W20260926 14:50:51.784575 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:51.779883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:38839 (local address 127.5.15.190:38653)
0926 14:50:51.779954 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:51.779956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:51.779963 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:50:51.779986 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:51.779987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:51.779997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:51.780044 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:51.780388 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.780853 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.781657 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.781850 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.781958 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.784510 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:38839: BlockingRecv error: recv got EOF from 127.5.15.130:38839 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 14:50:51 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:51 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:51.910036 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:51.905392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:37471)
0926 14:50:51.905531 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:51.905542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:51.905604 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:51.905739 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:51.905739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:51.905929 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:51.905971 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.905973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.906677 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.906678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.907287 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:51.907288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.907649 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.907650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:51.907709 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.908231 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:51.908236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:51.909972 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260926 14:50:51.910144 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:51.905523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:37471 (local address 127.5.15.190:38653)
0926 14:50:51.905580 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:51.905582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:51.905590 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:51.905657 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:51.905658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:51.905672 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:51.905724 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:51.906088 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.906586 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.907384 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:51.907569 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:51.907647 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:51.910071 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:37471: BlockingRecv error: recv got EOF from 127.5.15.131:37471 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 14:50:52 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:52 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:52.657373 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:52.652780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:54797)
0926 14:50:52.652913 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:52.652931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:52.652978 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:52.653113 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:52.653114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:52.653270 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:52.653325 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.653327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.653992 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.653993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.654770 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.654773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.655138 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.655139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.655193 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.655647 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:52.655652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:52.657315 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
W20260926 14:50:52.657485 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:52.652831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:54797 (local address 127.5.15.190:38653)
0926 14:50:52.652981 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:52.652982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:52.653014 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:52.653036 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:52.653037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:52.653047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:52.653096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:52.653406 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.653893 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.654873 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.655063 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.655140 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.657411 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:54797: BlockingRecv error: recv got EOF from 127.5.15.129:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
Sep 26 14:50:52 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:52 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:52.789337 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:52.785228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:38183)
0926 14:50:52.785394 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:52.785410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:52.785463 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:52.785584 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:52.785585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:52.785740 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:52.785788 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.785789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.786376 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.786376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.786904 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.786906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.787192 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.787193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.787228 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.787663 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:52.787667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:52.789282 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
W20260926 14:50:52.789434 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:52.785300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:38183 (local address 127.5.15.190:38653)
0926 14:50:52.785372 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:52.785374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:52.785463 (+ 89us) server_negotiation.cc:407] Connection header received
0926 14:50:52.785506 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:52.785507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:52.785515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:52.785564 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:52.785857 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.786313 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.786974 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.787134 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.787181 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.789371 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:38183: BlockingRecv error: recv got EOF from 127.5.15.130:38183 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 14:50:52 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:52 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.131@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:52.915133 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:52.910604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.131:54075)
0926 14:50:52.910783 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:52.910795 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:52.910867 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:52.911000 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:52.911000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:52.911180 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:52.911224 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.911226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.911903 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.911904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.912603 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:52.912605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.912906 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.912907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:52.912942 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.913411 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:52.913416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:52.915067 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260926 14:50:52.915251 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:52.910678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:54075 (local address 127.5.15.190:38653)
0926 14:50:52.910809 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:52.910813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:52.910853 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:50:52.910924 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:52.910924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:52.910936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:52.910983 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:52.911293 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.911827 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.912671 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:52.912847 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:52.912898 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:52.915155 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.131:54075: BlockingRecv error: recv got EOF from 127.5.15.131:54075 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 14:50:53 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:53 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.129@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:53.668080 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:53.657986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.129:52947)
0926 14:50:53.658130 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:53.658140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:53.658190 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:53.658318 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:53.658319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:53.658471 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:53.658519 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.658521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.659262 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.659263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.659771 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.659772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.660084 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.660085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.660123 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.666331 (+ 6208us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:53.666336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:53.668015 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
W20260926 14:50:53.668151 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:53.658061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:52947 (local address 127.5.15.190:38653)
0926 14:50:53.658182 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:53.658183 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:53.658191 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:50:53.658234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:53.658234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:53.658244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:53.658300 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:53.658583 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.659199 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.659844 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.660026 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.660072 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.668100 (+ 8028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.129:52947: BlockingRecv error: recv got EOF from 127.5.15.129:52947 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
Sep 26 14:50:53 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:50:53 dist-test-slave-s4p1 krb5kdc[13310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.15.130@KRBTEST.COM for oryx/127.5.15.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:50:53.794373 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:53.789876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38653 (local address 127.5.15.130:41319)
0926 14:50:53.790082 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:53.790094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:53.790184 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:53.790313 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:53.790313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:53.790490 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:53.790536 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.790538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.791278 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.791279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.791783 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:53.791784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.792195 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.792196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:53.792260 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.792677 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:53.792682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:53.794271 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.15.190:38653: Server oryx/127.5.15.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260926 14:50:53.794411 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:53.790000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:41319 (local address 127.5.15.190:38653)
0926 14:50:53.790079 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:53.790081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:53.790168 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:50:53.790241 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:53.790242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:53.790255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:53.790298 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:53.790614 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.791199 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.791894 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.792099 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:53.792193 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:53.794351 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.15.130:41319: BlockingRecv error: recv got EOF from 127.5.15.130:41319 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:50:53.843423 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13893
I20260926 14:50:53.847967 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14045
I20260926 14:50:53.852097 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14177
I20260926 14:50:53.856097 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 13823
2026-09-26T14:50:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17329 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T14:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:53Z Disabled control of system clock
I20260926 14:50:53.882562 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:32953
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:32953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:50:53.960167 14373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:53.960354 14373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:53.960372 14373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:53.961402 14373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:53.961434 14373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:53.961447 14373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:53.961458 14373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:53.961470 14373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:53.962654 14373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:32953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:32953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:53.962882 14373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:53.963095 14373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:53.967487 14378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:53.967528 14381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:53.967787 14373 server_base.cc:1034] running on GCE node
W20260926 14:50:53.967509 14379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:53.968115 14373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:53.968376 14373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:53.969516 14373 hybrid_clock.cc:648] HybridClock initialized: now 1790434253969509 us; error 29 us; skew 500 ppm
I20260926 14:50:53.970717 14373 webserver.cc:460] Webserver started at http://127.5.15.190:36955/ using document root <none> and password file <none>
I20260926 14:50:53.970957 14373 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:53.971000 14373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:53.971082 14373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:53.971975 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "2a15160a920c473586e44c29e68e6e3f"
format_stamp: "Formatted at 2026-09-26 14:50:53 on dist-test-slave-s4p1"
server_key: "eed85c9acb60036600698d9938e2b43b"
server_key_iv: "f42cfa40ea6fa49c102e7d03a7ef7061"
server_key_version: "encryptionkey@0"
I20260926 14:50:53.972247 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "2a15160a920c473586e44c29e68e6e3f"
format_stamp: "Formatted at 2026-09-26 14:50:53 on dist-test-slave-s4p1"
server_key: "eed85c9acb60036600698d9938e2b43b"
server_key_iv: "f42cfa40ea6fa49c102e7d03a7ef7061"
server_key_version: "encryptionkey@0"
I20260926 14:50:53.973347 14373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:50:53.974052 14387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:53.974202 14373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:53.974279 14373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "2a15160a920c473586e44c29e68e6e3f"
format_stamp: "Formatted at 2026-09-26 14:50:53 on dist-test-slave-s4p1"
server_key: "eed85c9acb60036600698d9938e2b43b"
server_key_iv: "f42cfa40ea6fa49c102e7d03a7ef7061"
server_key_version: "encryptionkey@0"
I20260926 14:50:53.974344 14373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:53.984467 14373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:53.984699 14373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:53.984836 14373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:53.988315 14373 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:32953
I20260926 14:50:53.988361 14439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:32953 every 8 connection(s)
I20260926 14:50:53.988667 14373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:53.989291 14440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:53.991616 14440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f: Bootstrap starting.
I20260926 14:50:53.992147 14440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:53.992419 14440 log.cc:824] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:53.993019 14440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f: No bootstrap required, opened a new log
I20260926 14:50:53.994508 14440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a15160a920c473586e44c29e68e6e3f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32953 } }
I20260926 14:50:53.994652 14440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:53.994697 14440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a15160a920c473586e44c29e68e6e3f, State: Initialized, Role: FOLLOWER
I20260926 14:50:53.994881 14440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a15160a920c473586e44c29e68e6e3f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32953 } }
I20260926 14:50:53.994980 14440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:53.995023 14440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:53.995070 14440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:53.995565 14440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a15160a920c473586e44c29e68e6e3f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32953 } }
I20260926 14:50:53.995680 14440 leader_election.cc:302] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a15160a920c473586e44c29e68e6e3f; no voters:
I20260926 14:50:53.995867 14440 leader_election.cc:288] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:53.995916 14443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:53.996104 14443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [term 1 LEADER]: Becoming Leader. State: Replica: 2a15160a920c473586e44c29e68e6e3f, State: Running, Role: LEADER
I20260926 14:50:53.996119 14440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:53.996198 14443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a15160a920c473586e44c29e68e6e3f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32953 } }
I20260926 14:50:53.996603 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14373
I20260926 14:50:53.996706 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:53.996929 5182 external_mini_cluster.cc:1428] Setting key c4f276b0e14a294c2a43a7b312c89e11
I20260926 14:50:53.997215 14445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a15160a920c473586e44c29e68e6e3f. Latest consensus state: current_term: 1 leader_uuid: "2a15160a920c473586e44c29e68e6e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a15160a920c473586e44c29e68e6e3f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32953 } } }
I20260926 14:50:53.997265 14444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a15160a920c473586e44c29e68e6e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a15160a920c473586e44c29e68e6e3f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32953 } } }
I20260926 14:50:53.997303 14445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:53.997308 14444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:53.997622 14459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:53.998253 14459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:53.999779 14459 catalog_manager.cc:1347] Generated new cluster ID: 5e4ec01d60894d96830824aafbd6d544
I20260926 14:50:53.999842 14459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:54.002898 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:53.997545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.5.15.190:32953)
0926 14:50:53.997741 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:53.997750 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:53.997807 (+ 57us) server_negotiation.cc:407] Connection header received
0926 14:50:53.997992 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:53.997999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:53.998073 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:53.998215 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:53.998427 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:53.999683 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.000789 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.001385 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.001508 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.002172 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.002176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.002176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.002370 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.002403 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.002414 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.002470 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.002561 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.002704 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260926 14:50:54.007661 14459 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:54.008289 14459 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:54.018343 14459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a15160a920c473586e44c29e68e6e3f: Generated new TSK 0
I20260926 14:50:54.018514 14459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:54.019711 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32953
--builtin_ntp_servers=127.5.15.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 14:50:54.102205 14464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.102411 14464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.102445 14464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.103572 14464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.103632 14464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.103670 14464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:54.104895 14464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.105126 14464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.105382 14464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.110826 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.110921 14464 server_base.cc:1034] running on GCE node
W20260926 14:50:54.110867 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.111002 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.111259 14464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.111545 14464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.112732 14464 hybrid_clock.cc:648] HybridClock initialized: now 1790434254112718 us; error 28 us; skew 500 ppm
I20260926 14:50:54.114073 14464 webserver.cc:460] Webserver started at http://127.5.15.129:45615/ using document root <none> and password file <none>
I20260926 14:50:54.114324 14464 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.114394 14464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.114528 14464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.115885 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "83d23d09db8c46948b7e9ec12eec960b"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "6a030a588c60de981cab37acc477d362"
server_key_iv: "45d06e6dbc0b8d260eb0524e3126d8b2"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.116278 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "83d23d09db8c46948b7e9ec12eec960b"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "6a030a588c60de981cab37acc477d362"
server_key_iv: "45d06e6dbc0b8d260eb0524e3126d8b2"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.117815 14464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:54.118592 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.118791 14464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:54.118889 14464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "83d23d09db8c46948b7e9ec12eec960b"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "6a030a588c60de981cab37acc477d362"
server_key_iv: "45d06e6dbc0b8d260eb0524e3126d8b2"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.118956 14464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.150020 14464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.150311 14464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.150456 14464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.150676 14464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:54.151033 14464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:54.151067 14464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.151090 14464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:54.151132 14464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.156670 14464 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:37763
I20260926 14:50:54.156718 14591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:37763 every 8 connection(s)
I20260926 14:50:54.157002 14464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:54.162017 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.157645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:50949 (local address 127.5.15.190:32953)
0926 14:50:54.157715 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.157717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.157984 (+ 267us) server_negotiation.cc:407] Connection header received
0926 14:50:54.158510 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.158512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.158526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.158579 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.159283 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.159766 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.160724 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.160939 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.161014 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.161727 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.161728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.161729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.161733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.161748 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.161752 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.161815 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.161912 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.161941 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:50:54.162245 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.157631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:32953 (local address 127.5.15.129:50949)
0926 14:50:54.157871 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.157903 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.158397 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.158663 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.158666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.158674 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.159184 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.159189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.159861 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.159862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.160622 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.160624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.161021 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.161022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.161150 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.161598 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:54.161599 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.161623 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:54.161642 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:54.161824 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:54.161830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:54.161832 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:54.161914 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:50:54.162008 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":123,"threads_started":1}
I20260926 14:50:54.162760 14592 heartbeater.cc:347] Connected to a master server at 127.5.15.190:32953
I20260926 14:50:54.162854 14592 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:54.162992 14592 heartbeater.cc:511] Master 127.5.15.190:32953 requested a full tablet report, sending...
I20260926 14:50:54.163448 14404 ts_manager.cc:194] Registered new tserver with Master: 83d23d09db8c46948b7e9ec12eec960b (127.5.15.129:37763)
I20260926 14:50:54.164011 14404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.129:50949
I20260926 14:50:54.165758 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14464
I20260926 14:50:54.165831 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:54.166033 5182 external_mini_cluster.cc:1428] Setting key 40292072a64af4b236811d86ee5df948
I20260926 14:50:54.166869 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32953
--builtin_ntp_servers=127.5.15.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 14:50:54.245481 14595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.245689 14595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.245724 14595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.247018 14595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.247073 14595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.247112 14595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:54.248397 14595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.248651 14595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.248924 14595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.253577 14600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.253631 14601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.253680 14595 server_base.cc:1034] running on GCE node
W20260926 14:50:54.253597 14603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.254029 14595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.254269 14595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.255436 14595 hybrid_clock.cc:648] HybridClock initialized: now 1790434254255414 us; error 33 us; skew 500 ppm
I20260926 14:50:54.256590 14595 webserver.cc:460] Webserver started at http://127.5.15.130:34811/ using document root <none> and password file <none>
I20260926 14:50:54.256811 14595 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.256876 14595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.257019 14595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.258005 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "1a5d9f448f044f71b1d08a368b2bbf4b"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "6a739acb67fb915405d22348b0a0ffc5"
server_key_iv: "989e0000b2ee5ebf4bb430b0a690bb56"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.258350 14595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "1a5d9f448f044f71b1d08a368b2bbf4b"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "6a739acb67fb915405d22348b0a0ffc5"
server_key_iv: "989e0000b2ee5ebf4bb430b0a690bb56"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.259639 14595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:54.260315 14609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.260597 14595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:54.260690 14595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "1a5d9f448f044f71b1d08a368b2bbf4b"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "6a739acb67fb915405d22348b0a0ffc5"
server_key_iv: "989e0000b2ee5ebf4bb430b0a690bb56"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.260766 14595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.282892 14595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.283140 14595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.283275 14595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.283497 14595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:54.283777 14595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:54.283829 14595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.283869 14595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:54.283895 14595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.289255 14595 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:36547
I20260926 14:50:54.289330 14722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:36547 every 8 connection(s)
I20260926 14:50:54.289635 14595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:54.291218 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14595
I20260926 14:50:54.291316 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:54.291529 5182 external_mini_cluster.cc:1428] Setting key 4059b0e14dd1bb7e2ff809629a8ad5ef
I20260926 14:50:54.292450 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32953
--builtin_ntp_servers=127.5.15.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:50:54.295706 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.290301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:47497 (local address 127.5.15.190:32953)
0926 14:50:54.290370 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.290372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.290637 (+ 265us) server_negotiation.cc:407] Connection header received
0926 14:50:54.291196 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.291197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.291206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.291245 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.292115 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.292708 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.294139 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.294444 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.294536 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.295394 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.295395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.295396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.295400 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.295416 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.295421 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.295498 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.295560 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.295590 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:50:54.295876 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.290301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:32953 (local address 127.5.15.130:47497)
0926 14:50:54.290489 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.290545 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.291094 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.291325 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.291329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.291339 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.292024 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.292029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.292799 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.292801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.294058 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.294061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.294546 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.294547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.294647 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.295238 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:54.295239 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.295271 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:54.295291 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:54.295505 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:54.295510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:54.295512 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:54.295560 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:50:54.295633 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260926 14:50:54.296348 14723 heartbeater.cc:347] Connected to a master server at 127.5.15.190:32953
I20260926 14:50:54.296422 14723 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:54.296646 14723 heartbeater.cc:511] Master 127.5.15.190:32953 requested a full tablet report, sending...
I20260926 14:50:54.297025 14404 ts_manager.cc:194] Registered new tserver with Master: 1a5d9f448f044f71b1d08a368b2bbf4b (127.5.15.130:36547)
I20260926 14:50:54.297410 14404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.130:47497
W20260926 14:50:54.374670 14726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.374922 14726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.374957 14726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.376125 14726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.376184 14726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.376224 14726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:54.377583 14726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.377837 14726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.378118 14726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.382934 14734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.382915 14731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.382915 14732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.383006 14726 server_base.cc:1034] running on GCE node
I20260926 14:50:54.383291 14726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.383548 14726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.384707 14726 hybrid_clock.cc:648] HybridClock initialized: now 1790434254384702 us; error 34 us; skew 500 ppm
I20260926 14:50:54.385905 14726 webserver.cc:460] Webserver started at http://127.5.15.131:43271/ using document root <none> and password file <none>
I20260926 14:50:54.386147 14726 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.386211 14726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.386339 14726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.387444 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "d114afa84b434a1880b59054a439cae8"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "36b89ad2c84920aecae1c1d783217e30"
server_key_iv: "07cbc6e2bd365e6fed3374e935a0a008"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.387794 14726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "d114afa84b434a1880b59054a439cae8"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "36b89ad2c84920aecae1c1d783217e30"
server_key_iv: "07cbc6e2bd365e6fed3374e935a0a008"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.389070 14726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:50:54.389750 14740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.389927 14726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:54.390033 14726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "d114afa84b434a1880b59054a439cae8"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "36b89ad2c84920aecae1c1d783217e30"
server_key_iv: "07cbc6e2bd365e6fed3374e935a0a008"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.390116 14726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.414168 14726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.414453 14726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.414590 14726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.414845 14726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:54.415150 14726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:54.415203 14726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.415232 14726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:54.415261 14726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.420815 14726 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:34085
I20260926 14:50:54.420857 14853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:34085 every 8 connection(s)
I20260926 14:50:54.421185 14726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:54.425751 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.421835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:34497 (local address 127.5.15.190:32953)
0926 14:50:54.421909 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.421911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.422160 (+ 249us) server_negotiation.cc:407] Connection header received
0926 14:50:54.422594 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.422595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.422604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.422651 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.423255 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.423747 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.424616 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.424785 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.424832 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.425540 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.425541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.425542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.425546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.425559 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.425563 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.425609 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.425671 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.425693 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:54.425915 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.421806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:32953 (local address 127.5.15.131:34497)
0926 14:50:54.422032 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.422072 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.422519 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.422706 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.422709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.422717 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.423190 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.423193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.423835 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.423835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.424549 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:54.424551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:54.424871 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:54.424871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:54.424973 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:54.425415 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:54.425415 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.425440 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:54.425459 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:54.425611 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:54.425613 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:54.425616 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:54.425659 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:50:54.425727 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260926 14:50:54.426371 14854 heartbeater.cc:347] Connected to a master server at 127.5.15.190:32953
I20260926 14:50:54.426465 14854 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:54.426630 14854 heartbeater.cc:511] Master 127.5.15.190:32953 requested a full tablet report, sending...
I20260926 14:50:54.427052 14404 ts_manager.cc:194] Registered new tserver with Master: d114afa84b434a1880b59054a439cae8 (127.5.15.131:34085)
I20260926 14:50:54.427420 14404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.131:34497
I20260926 14:50:54.427863 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14726
I20260926 14:50:54.427938 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:54.428117 5182 external_mini_cluster.cc:1428] Setting key 1c92b0f8e2630a84e0cbebfda90b541a
I20260926 14:50:54.428958 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:50:54.430645 14461 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:54.430220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.5.15.190:32953)
0926 14:50:54.430283 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.430284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.430407 (+ 123us) server_negotiation.cc:407] Connection header received
0926 14:50:54.430453 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.430454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.430463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.430500 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.430606 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40158: BlockingRecv error: recv got EOF from 127.0.0.1:40158 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:50:54.430725 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14464
I20260926 14:50:54.435386 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14595
I20260926 14:50:54.439599 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14726
I20260926 14:50:54.445441 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14373
2026-09-26T14:50:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (588 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T14:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:54Z Disabled control of system clock
I20260926 14:50:54.471094 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44275
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:44275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:50:54.547968 14865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.548141 14865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.548161 14865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.549229 14865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:54.549261 14865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.549274 14865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.549286 14865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:54.549299 14865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:54.550491 14865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:44275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.550668 14865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.550905 14865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.555322 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.555459 14865 server_base.cc:1034] running on GCE node
W20260926 14:50:54.555337 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.555337 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.555819 14865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.556082 14865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.557214 14865 hybrid_clock.cc:648] HybridClock initialized: now 1790434254557192 us; error 39 us; skew 500 ppm
I20260926 14:50:54.558349 14865 webserver.cc:460] Webserver started at http://127.5.15.190:41383/ using document root <none> and password file <none>
I20260926 14:50:54.558583 14865 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.558625 14865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.558825 14865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.559787 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "cc8f0f2ef6eb4067a8b254da2273b18f"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "ddf59e63ffd65b9bbd1a3bae6427b30b"
server_key_iv: "1e179e6e0e3fb7890b5118624cfecfc0"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.560139 14865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "cc8f0f2ef6eb4067a8b254da2273b18f"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "ddf59e63ffd65b9bbd1a3bae6427b30b"
server_key_iv: "1e179e6e0e3fb7890b5118624cfecfc0"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.561306 14865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:50:54.561961 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.562132 14865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:54.562228 14865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "cc8f0f2ef6eb4067a8b254da2273b18f"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "ddf59e63ffd65b9bbd1a3bae6427b30b"
server_key_iv: "1e179e6e0e3fb7890b5118624cfecfc0"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.562311 14865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.565595 14865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.565860 14865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.566015 14865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.569669 14865 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:44275
I20260926 14:50:54.569697 14931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:44275 every 8 connection(s)
I20260926 14:50:54.569998 14865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:54.570574 14932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:54.573046 14932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f: Bootstrap starting.
I20260926 14:50:54.573585 14932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:54.573865 14932 log.cc:824] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:54.574463 14932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f: No bootstrap required, opened a new log
I20260926 14:50:54.574692 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14865
I20260926 14:50:54.574827 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:54.574990 5182 external_mini_cluster.cc:1428] Setting key f7dfb449d5fc71b1973011844e0d9921
I20260926 14:50:54.576682 14932 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44275 } }
I20260926 14:50:54.576834 14932 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:54.576898 14932 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8f0f2ef6eb4067a8b254da2273b18f, State: Initialized, Role: FOLLOWER
I20260926 14:50:54.577014 14932 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44275 } }
I20260926 14:50:54.577105 14932 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:54.577139 14932 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:54.577190 14932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:54.577527 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.575432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.5.15.190:44275)
0926 14:50:54.575627 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.575636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.575668 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:50:54.576242 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.576250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.576299 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.576459 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.577051 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.577054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.577054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.577059 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.577101 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.577115 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.577184 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.577291 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.577350 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260926 14:50:54.578205 14932 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44275 } }
I20260926 14:50:54.578315 14932 leader_election.cc:302] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8f0f2ef6eb4067a8b254da2273b18f; no voters:
I20260926 14:50:54.578478 14932 leader_election.cc:288] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:54.578500 14937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:54.578629 14937 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [term 1 LEADER]: Becoming Leader. State: Replica: cc8f0f2ef6eb4067a8b254da2273b18f, State: Running, Role: LEADER
I20260926 14:50:54.578709 14932 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:54.578788 14937 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44275 } }
I20260926 14:50:54.579196 14939 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8f0f2ef6eb4067a8b254da2273b18f. Latest consensus state: current_term: 1 leader_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44275 } } }
I20260926 14:50:54.579197 14938 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8f0f2ef6eb4067a8b254da2273b18f" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44275 } } }
I20260926 14:50:54.579344 14939 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:54.579381 14938 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:54.579542 14946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:54.580018 14946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:54.581429 14946 catalog_manager.cc:1347] Generated new cluster ID: 9756aaaef8834fc295a900cd7fa0bf58
I20260926 14:50:54.581480 14946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:50:54.594434 14946 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:54.594590 14946 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:54.604873 14946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc8f0f2ef6eb4067a8b254da2273b18f: Generated new TSK 0
I20260926 14:50:54.605036 14946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:50:54.610641 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44275
--builtin_ntp_servers=127.5.15.148:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:50:54.689567 14956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.689747 14956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.689769 14956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.690966 14956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.691004 14956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.691028 14956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:54.692315 14956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.692534 14956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.692818 14956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.698452 14962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.698439 14961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.698633 14956 server_base.cc:1034] running on GCE node
W20260926 14:50:54.698678 14964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.698858 14956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.699110 14956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.700265 14956 hybrid_clock.cc:648] HybridClock initialized: now 1790434254700255 us; error 27 us; skew 500 ppm
I20260926 14:50:54.701301 14956 webserver.cc:460] Webserver started at http://127.5.15.129:36087/ using document root <none> and password file <none>
I20260926 14:50:54.701529 14956 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.701573 14956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.701717 14956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.702701 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "76088bc0509f4972ac39914191955a9a"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "80be5214d3969bae3cf39207c35e0b0b"
server_key_iv: "2ad4b42c49fda124dff49725ae92e2a5"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.703099 14956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "76088bc0509f4972ac39914191955a9a"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "80be5214d3969bae3cf39207c35e0b0b"
server_key_iv: "2ad4b42c49fda124dff49725ae92e2a5"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.704303 14956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:50:54.704986 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.705194 14956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:54.705260 14956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "76088bc0509f4972ac39914191955a9a"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "80be5214d3969bae3cf39207c35e0b0b"
server_key_iv: "2ad4b42c49fda124dff49725ae92e2a5"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.705343 14956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.708523 14956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.708767 14956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.708940 14956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.709142 14956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:54.709443 14956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:54.709475 14956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.709528 14956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:54.709554 14956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.715137 14956 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:34641
I20260926 14:50:54.715171 15083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:34641 every 8 connection(s)
I20260926 14:50:54.715483 14956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:54.718207 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.716174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:42127 (local address 127.5.15.190:44275)
0926 14:50:54.716214 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.716216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.716446 (+ 230us) server_negotiation.cc:407] Connection header received
0926 14:50:54.716856 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.716857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.716861 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.716914 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.717994 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.717995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.717996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.717999 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.718013 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.718017 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.718064 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.718144 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.718156 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 14:50:54.718379 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.716143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44275 (local address 127.5.15.129:42127)
0926 14:50:54.716322 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.716360 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.716780 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.716971 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.716974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.716982 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.717856 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:54.717856 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.717890 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:54.717917 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:54.718076 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:54.718080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:54.718082 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:54.718141 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:50:54.718189 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260926 14:50:54.718890 15084 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44275
I20260926 14:50:54.718984 15084 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:54.719129 15084 heartbeater.cc:511] Master 127.5.15.190:44275 requested a full tablet report, sending...
I20260926 14:50:54.719609 14896 ts_manager.cc:194] Registered new tserver with Master: 76088bc0509f4972ac39914191955a9a (127.5.15.129:34641)
I20260926 14:50:54.725131 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 14956
I20260926 14:50:54.725203 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:54.725412 5182 external_mini_cluster.cc:1428] Setting key aa94783ef9bcb18416d9b82de9742121
I20260926 14:50:54.726277 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44275
--builtin_ntp_servers=127.5.15.148:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:50:54.805509 15087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.805692 15087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.805713 15087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.806913 15087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.806952 15087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.806977 15087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:54.808352 15087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.808588 15087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.808871 15087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.813536 15093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.813624 15092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.813540 15095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.813815 15087 server_base.cc:1034] running on GCE node
I20260926 14:50:54.814034 15087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.814270 15087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.815425 15087 hybrid_clock.cc:648] HybridClock initialized: now 1790434254815421 us; error 34 us; skew 500 ppm
I20260926 14:50:54.816702 15087 webserver.cc:460] Webserver started at http://127.5.15.130:46265/ using document root <none> and password file <none>
I20260926 14:50:54.816947 15087 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.817013 15087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.817147 15087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.818163 15087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "7b619823bd124603b3e67af86c7f281c"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "dd50d25c4437ad468817e74e9bd3ed25"
server_key_iv: "045ee8e0feb8a7b0922c8b9e303ca34d"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.818529 15087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "7b619823bd124603b3e67af86c7f281c"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "dd50d25c4437ad468817e74e9bd3ed25"
server_key_iv: "045ee8e0feb8a7b0922c8b9e303ca34d"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.819823 15087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:50:54.820529 15101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.820700 15087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:54.820796 15087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "7b619823bd124603b3e67af86c7f281c"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "dd50d25c4437ad468817e74e9bd3ed25"
server_key_iv: "045ee8e0feb8a7b0922c8b9e303ca34d"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.820883 15087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.824153 15087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.824406 15087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.824539 15087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.824745 15087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:54.825027 15087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:54.825080 15087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.825117 15087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:54.825143 15087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.830725 15087 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:38499
I20260926 14:50:54.830801 15214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:38499 every 8 connection(s)
I20260926 14:50:54.831105 15087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:54.833653 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.831756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:35571 (local address 127.5.15.190:44275)
0926 14:50:54.831810 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.831813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.832042 (+ 229us) server_negotiation.cc:407] Connection header received
0926 14:50:54.832435 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.832437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.832442 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.832492 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.833480 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.833481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.833481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.833485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.833496 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.833500 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.833543 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.833591 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.833602 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:54.833853 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.831746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44275 (local address 127.5.15.130:35571)
0926 14:50:54.831928 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.831958 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.832351 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.832539 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.832542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.832549 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.833348 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:54.833349 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.833381 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:54.833406 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:54.833542 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:54.833544 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:54.833546 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:50:54.833591 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:50:54.833633 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260926 14:50:54.834256 15215 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44275
I20260926 14:50:54.834323 15215 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:54.834467 15215 heartbeater.cc:511] Master 127.5.15.190:44275 requested a full tablet report, sending...
I20260926 14:50:54.834853 14896 ts_manager.cc:194] Registered new tserver with Master: 7b619823bd124603b3e67af86c7f281c (127.5.15.130:38499)
I20260926 14:50:54.840641 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 15087
I20260926 14:50:54.840741 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:54.840914 5182 external_mini_cluster.cc:1428] Setting key f77af8766e1d876ca23dcd64b1f9c70f
I20260926 14:50:54.841728 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44275
--builtin_ntp_servers=127.5.15.148:34305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:50:54.920090 15218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:54.920262 15218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:54.920282 15218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:54.921358 15218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:54.921396 15218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:54.921420 15218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:54.922667 15218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:54.922914 15218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:54.923183 15218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:54.927616 15226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.927695 15224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:54.927625 15223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:54.927870 15218 server_base.cc:1034] running on GCE node
I20260926 14:50:54.928082 15218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:54.928344 15218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:54.929507 15218 hybrid_clock.cc:648] HybridClock initialized: now 1790434254929500 us; error 32 us; skew 500 ppm
I20260926 14:50:54.930820 15218 webserver.cc:460] Webserver started at http://127.5.15.131:45317/ using document root <none> and password file <none>
I20260926 14:50:54.931054 15218 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:54.931118 15218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:54.931245 15218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:54.932189 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "f4470bac04b045a1b0d447e4a505d281"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "aae75e38b07b8d97769bfd6c5d6786c9"
server_key_iv: "13b5be0d8ebfc029848429baa265aabd"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.932539 15218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "f4470bac04b045a1b0d447e4a505d281"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "aae75e38b07b8d97769bfd6c5d6786c9"
server_key_iv: "13b5be0d8ebfc029848429baa265aabd"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.933724 15218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:54.934346 15232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.934521 15218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:54.934612 15218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "f4470bac04b045a1b0d447e4a505d281"
format_stamp: "Formatted at 2026-09-26 14:50:54 on dist-test-slave-s4p1"
server_key: "aae75e38b07b8d97769bfd6c5d6786c9"
server_key_iv: "13b5be0d8ebfc029848429baa265aabd"
server_key_version: "encryptionkey@0"
I20260926 14:50:54.934693 15218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:54.937847 15218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:54.938086 15218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:54.938216 15218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:54.938424 15218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:54.938707 15218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:54.938776 15218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.938808 15218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:54.938839 15218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:54.944450 15218 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46621
I20260926 14:50:54.944497 15345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46621 every 8 connection(s)
I20260926 14:50:54.944773 15218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:54.945909 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 15218
I20260926 14:50:54.946007 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:54.946192 5182 external_mini_cluster.cc:1428] Setting key 80cd74129a51a7bd5cb1d746774dace3
I20260926 14:50:54.947811 14936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.945391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38995 (local address 127.5.15.190:44275)
0926 14:50:54.945431 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.945433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.945674 (+ 241us) server_negotiation.cc:407] Connection header received
0926 14:50:54.946194 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.946195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.946202 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.946263 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.947559 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:54.947560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:54.947561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:50:54.947565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:54.947580 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:50:54.947584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:54.947655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:54.947731 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 14:50:54.947746 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:54.947992 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:54.945388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44275 (local address 127.5.15.131:38995)
0926 14:50:54.945562 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:54.945590 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:54.946085 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:54.946328 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:54.946332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:54.946341 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:54.947406 (+ 1065us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:50:54.947407 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:54.947433 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:50:54.947458 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:54.947653 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:54.947659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:54.947662 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:50:54.947729 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:50:54.947787 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260926 14:50:54.948414 15346 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44275
I20260926 14:50:54.948488 15346 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:54.948628 15346 heartbeater.cc:511] Master 127.5.15.190:44275 requested a full tablet report, sending...
I20260926 14:50:54.948982 14896 ts_manager.cc:194] Registered new tserver with Master: f4470bac04b045a1b0d447e4a505d281 (127.5.15.131:46621)
I20260926 14:50:54.958122 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:50:54.959933 14936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:50:54.959483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.5.15.190:44275)
0926 14:50:54.959556 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:54.959558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:54.959627 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:50:54.959697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:54.959698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:54.959702 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:54.959757 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:54.959873 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35256: BlockingRecv error: recv got EOF from 127.0.0.1:35256 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:50:54.960000 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14956
I20260926 14:50:54.964151 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 15087
I20260926 14:50:54.968227 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 15218
I20260926 14:50:54.972249 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 14865
2026-09-26T14:50:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (526 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:55 dist-test-slave-s4p1 krb5kdc[15356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:55 dist-test-slave-s4p1 krb5kdc[15356](info): set up 2 sockets
Sep 26 14:50:55 dist-test-slave-s4p1 krb5kdc[15356](info): commencing operation
krb5kdc: starting...
W20260926 14:50:55.991592 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:50:56.103670 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41705
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:56.180660 15369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:56.180882 15369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:56.180917 15369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:56.181986 15369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:50:56.182070 15369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:56.182094 15369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:56.182125 15369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:50:56.182147 15369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:50:56.183698 15369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:41705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:56.183950 15369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:56.184180 15369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:56.188771 15374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.188829 15375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.188786 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:56.188954 15369 server_base.cc:1034] running on GCE node
I20260926 14:50:56.189250 15369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:56.189527 15369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:56.190693 15369 hybrid_clock.cc:648] HybridClock initialized: now 1790434256190685 us; error 36 us; skew 500 ppm
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:56.193171 15369 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:50:56.193532 15369 webserver.cc:460] Webserver started at http://127.5.15.190:34811/ using document root <none> and password file <none>
I20260926 14:50:56.193727 15369 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:56.193766 15369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:56.193847 15369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:56.194885 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "64547eb69bad4df08f4b3dacf46fec2e"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "8c2764cc975082ce5b20e40c07afcacc"
server_key_iv: "b7ef8bbbb37687a5945decd98e339fa6"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.195178 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "64547eb69bad4df08f4b3dacf46fec2e"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "8c2764cc975082ce5b20e40c07afcacc"
server_key_iv: "b7ef8bbbb37687a5945decd98e339fa6"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.196365 15369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:56.197062 15384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.197243 15369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:50:56.197324 15369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "64547eb69bad4df08f4b3dacf46fec2e"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "8c2764cc975082ce5b20e40c07afcacc"
server_key_iv: "b7ef8bbbb37687a5945decd98e339fa6"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.197382 15369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:56.212563 15369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:56.214325 15369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:56.214480 15369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:56.218356 15369 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:41705
I20260926 14:50:56.218410 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:41705 every 8 connection(s)
I20260926 14:50:56.218825 15369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:50:56.219800 15437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:56.221889 15437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e: Bootstrap starting.
I20260926 14:50:56.222394 15437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:56.222609 15437 log.cc:824] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:56.223316 15437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e: No bootstrap required, opened a new log
I20260926 14:50:56.224772 15437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64547eb69bad4df08f4b3dacf46fec2e" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41705 } }
I20260926 14:50:56.224910 15437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:56.224931 15437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64547eb69bad4df08f4b3dacf46fec2e, State: Initialized, Role: FOLLOWER
I20260926 14:50:56.225057 15437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64547eb69bad4df08f4b3dacf46fec2e" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41705 } }
I20260926 14:50:56.225135 15437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:50:56.225157 15437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:50:56.225185 15437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:56.225894 15437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64547eb69bad4df08f4b3dacf46fec2e" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41705 } }
I20260926 14:50:56.225976 15437 leader_election.cc:302] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64547eb69bad4df08f4b3dacf46fec2e; no voters:
I20260926 14:50:56.226090 15437 leader_election.cc:288] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:50:56.226155 15440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:56.226348 15440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [term 1 LEADER]: Becoming Leader. State: Replica: 64547eb69bad4df08f4b3dacf46fec2e, State: Running, Role: LEADER
I20260926 14:50:56.226405 15437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:50:56.226437 15440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64547eb69bad4df08f4b3dacf46fec2e" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41705 } }
I20260926 14:50:56.226876 15442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64547eb69bad4df08f4b3dacf46fec2e. Latest consensus state: current_term: 1 leader_uuid: "64547eb69bad4df08f4b3dacf46fec2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64547eb69bad4df08f4b3dacf46fec2e" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41705 } } }
I20260926 14:50:56.226858 15441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64547eb69bad4df08f4b3dacf46fec2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64547eb69bad4df08f4b3dacf46fec2e" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41705 } } }
I20260926 14:50:56.226955 15442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:56.226962 15441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e [sys.catalog]: This master's current role is: LEADER
I20260926 14:50:56.227417 15448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:50:56.227909 15448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:50:56.227901 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 15369
I20260926 14:50:56.227998 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:50:56.228214 5182 external_mini_cluster.cc:1428] Setting key a60d4ee6bd7aa8e4710ace262d85e0e6
I20260926 14:50:56.229434 15448 catalog_manager.cc:1347] Generated new cluster ID: 6aad37641db749f1ac4fe7eddb271c30
I20260926 14:50:56.229478 15448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:56.237973 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.228696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.5.15.190:41705)
0926 14:50:56.229089 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.229097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.229131 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:50:56.229219 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.229224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.229264 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.229404 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.230023 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.231006 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.231800 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.232258 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.232392 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.234669 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:56.234673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:56.234673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:56.234687 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:56.236395 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.236757 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.236760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.236761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.236895 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.237103 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.237104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.237104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.237331 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:56.237504 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.237570 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.237817 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":74,"threads_started":1}
I20260926 14:50:56.244477 15448 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:50:56.244971 15448 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:50:56.252918 15448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e: Generated new TSK 0
I20260926 14:50:56.253108 15448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:50:56.283747 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41705
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:56.400110 15463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:56.400386 15463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:56.400417 15463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:56.402089 15463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:56.402143 15463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:56.402179 15463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:50:56.404284 15463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:56.404525 15463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:56.404806 15463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:56.410054 15468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.410056 15471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.410056 15469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:56.410135 15463 server_base.cc:1034] running on GCE node
I20260926 14:50:56.410478 15463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:56.410730 15463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:56.411914 15463 hybrid_clock.cc:648] HybridClock initialized: now 1790434256411909 us; error 33 us; skew 500 ppm
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:56.414340 15463 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:50:56.414693 15463 webserver.cc:460] Webserver started at http://127.5.15.129:44943/ using document root <none> and password file <none>
I20260926 14:50:56.414930 15463 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:56.414975 15463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:56.415062 15463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:56.416013 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "acbfe634e043a322be47a97f9bef63fc"
server_key_iv: "79e9d4ade45cb47e58dfbc579f739f9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.416303 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "acbfe634e043a322be47a97f9bef63fc"
server_key_iv: "79e9d4ade45cb47e58dfbc579f739f9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.417495 15463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 14:50:56.418149 15478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.418299 15463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:56.418385 15463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "acbfe634e043a322be47a97f9bef63fc"
server_key_iv: "79e9d4ade45cb47e58dfbc579f739f9c"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.418438 15463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:56.430939 15463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:56.432627 15463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:56.432731 15463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:56.432897 15463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:56.433132 15463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:56.433161 15463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.433182 15463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:56.433193 15463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.438510 15463 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:38911
I20260926 14:50:56.438583 15591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:38911 every 8 connection(s)
I20260926 14:50:56.438934 15463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:50:56.439189 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 15463
I20260926 14:50:56.439289 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:50:56.439515 5182 external_mini_cluster.cc:1428] Setting key 8695cc1eca698908946d8355b1c549d6
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
I20260926 14:50:56.449309 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.439813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:36305 (local address 127.5.15.190:41705)
0926 14:50:56.439858 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.439861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.439978 (+ 117us) server_negotiation.cc:407] Connection header received
0926 14:50:56.440566 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.440567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.440583 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.440648 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.441764 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.442258 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.443818 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.444011 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.444065 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.446450 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:56.446451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:56.446452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:56.446461 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:56.447941 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.448340 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.448341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.448342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.448402 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.448654 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.448655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.448655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.448770 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:56.448855 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.449131 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.449216 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:50:56.449506 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.439656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41705 (local address 127.5.15.129:36305)
0926 14:50:56.439845 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.439901 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.440459 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.440717 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.440721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.441366 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:56.441658 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.441665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.442334 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.442335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.443733 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.443737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.444099 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.444101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.444251 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.444761 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:56.444772 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:56.446352 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:56.448020 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:56.448027 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:56.448029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:56.448251 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:56.448481 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:56.448482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:56.448482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:56.448585 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:56.448846 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:56.448852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:56.448994 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.449109 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.449263 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
I20260926 14:50:56.450045 15592 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41705
I20260926 14:50:56.450146 15592 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:56.450338 15592 heartbeater.cc:511] Master 127.5.15.190:41705 requested a full tablet report, sending...
I20260926 14:50:56.450970 15401 ts_manager.cc:194] Registered new tserver with Master: 4a17d2d371a743c88e7ec4dd3c40dd4e (127.5.15.129:38911)
I20260926 14:50:56.451742 15401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:36305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:50:56.464830 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41705
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:56.547066 15597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:56.547273 15597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:56.547295 15597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:56.548432 15597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:56.548473 15597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:56.548499 15597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:50:56.549935 15597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:56.550164 15597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:56.550395 15597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:50:56.555246 15603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.555244 15602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.555310 15605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:56.555392 15597 server_base.cc:1034] running on GCE node
I20260926 14:50:56.555687 15597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:56.555943 15597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:56.557101 15597 hybrid_clock.cc:648] HybridClock initialized: now 1790434256557088 us; error 44 us; skew 500 ppm
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:56.559537 15597 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:50:56.560000 15597 webserver.cc:460] Webserver started at http://127.5.15.130:33319/ using document root <none> and password file <none>
I20260926 14:50:56.560222 15597 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:56.560287 15597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:56.560434 15597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:56.561439 15597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "35728e253e25fb70f56666488a67ae58"
server_key_iv: "8c1456e85359dac6046aeb07193784cd"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.561810 15597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "35728e253e25fb70f56666488a67ae58"
server_key_iv: "8c1456e85359dac6046aeb07193784cd"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.563144 15597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 14:50:56.563850 15612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.564013 15597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:50:56.564107 15597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "35728e253e25fb70f56666488a67ae58"
server_key_iv: "8c1456e85359dac6046aeb07193784cd"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.564186 15597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:56.596959 15597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:56.598913 15597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:56.599074 15597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:56.599352 15597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:56.599676 15597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:56.599730 15597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.599771 15597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:56.599797 15597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.605461 15597 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:33403
I20260926 14:50:56.605499 15725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:33403 every 8 connection(s)
I20260926 14:50:56.605829 15597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:50:56.609532 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 15597
I20260926 14:50:56.609637 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:50:56.609824 5182 external_mini_cluster.cc:1428] Setting key 1f58a40f140fd15adf4c4c62a04d8472
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:56.615381 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.606490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:47027 (local address 127.5.15.190:41705)
0926 14:50:56.606568 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.606570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.606822 (+ 252us) server_negotiation.cc:407] Connection header received
0926 14:50:56.607363 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.607365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.607376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.607444 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.608374 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.608887 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.609988 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.610268 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.610350 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.612517 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:56.612519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:56.612519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:56.612527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:56.613919 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.614348 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.614350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.614350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.614418 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.614669 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.614670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.614670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.614839 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:56.614928 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.615194 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.615280 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:50:56.615573 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.606482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41705 (local address 127.5.15.130:47027)
0926 14:50:56.606663 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.606714 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.607245 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.607515 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.607520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.608082 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:56.608261 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.608266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.608980 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.608981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.609916 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.609919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.610368 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.610368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.610499 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.611092 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:56.611100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:56.612422 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:56.614022 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:56.614028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:56.614030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:56.614255 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:56.614489 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:56.614489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:56.614490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:56.614600 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:56.614917 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:56.614923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:56.615062 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.615173 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.615323 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260926 14:50:56.616078 15726 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41705
I20260926 14:50:56.616180 15726 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:56.616364 15726 heartbeater.cc:511] Master 127.5.15.190:41705 requested a full tablet report, sending...
I20260926 14:50:56.616737 15401 ts_manager.cc:194] Registered new tserver with Master: cfeb5fdda1494a4cbe2b9d3d812a3a57 (127.5.15.130:33403)
I20260926 14:50:56.617100 15401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:47027
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:50:56.633694 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41705
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:50:56.711270 15731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:50:56.711480 15731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:50:56.711503 15731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:50:56.712621 15731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:50:56.712661 15731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:50:56.712718 15731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:50:56.714113 15731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:50:56.714294 15731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:50:56.714522 15731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:50:56.719453 15731 server_base.cc:1034] running on GCE node
W20260926 14:50:56.719409 15736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.719448 15737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:50:56.719448 15739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:50:56.719724 15731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:50:56.719980 15731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:50:56.721151 15731 hybrid_clock.cc:648] HybridClock initialized: now 1790434256721136 us; error 26 us; skew 500 ppm
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:50:56.723788 15731 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:50:56.724211 15731 webserver.cc:460] Webserver started at http://127.5.15.131:40185/ using document root <none> and password file <none>
I20260926 14:50:56.724428 15731 fs_manager.cc:359] Metadata directory not provided
I20260926 14:50:56.724467 15731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:50:56.724550 15731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:50:56.725455 15731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "0bbef3af502c43ebb11a1de4c743989f"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "df9b8731449c52f178e5a26174ae3a97"
server_key_iv: "50e6c8b75a3c700c0fa4e4b1c92c123e"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.725739 15731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "0bbef3af502c43ebb11a1de4c743989f"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "df9b8731449c52f178e5a26174ae3a97"
server_key_iv: "50e6c8b75a3c700c0fa4e4b1c92c123e"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.727190 15731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:50:56.728021 15746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.728222 15731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:50:56.728315 15731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "0bbef3af502c43ebb11a1de4c743989f"
format_stamp: "Formatted at 2026-09-26 14:50:56 on dist-test-slave-s4p1"
server_key: "df9b8731449c52f178e5a26174ae3a97"
server_key_iv: "50e6c8b75a3c700c0fa4e4b1c92c123e"
server_key_version: "encryptionkey@0"
I20260926 14:50:56.728391 15731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:50:56.756732 15731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:50:56.758981 15731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:50:56.759090 15731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:50:56.759248 15731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:50:56.759477 15731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:50:56.759505 15731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.759526 15731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:50:56.759538 15731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:50:56.765118 15731 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:44241
I20260926 14:50:56.765162 15859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:44241 every 8 connection(s)
I20260926 14:50:56.765447 15731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:50:56.768087 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 15731
I20260926 14:50:56.768172 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:50:56.768337 5182 external_mini_cluster.cc:1428] Setting key f5b1ad1b6eb678db52cf884b5e8410bd
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434256, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:50:56.774480 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.766092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:59115 (local address 127.5.15.190:41705)
0926 14:50:56.766168 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.766170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.766340 (+ 170us) server_negotiation.cc:407] Connection header received
0926 14:50:56.766864 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.766865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.766878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.766932 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.767755 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.768512 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.769635 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.769828 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.769881 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.771815 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:56.771816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:56.771817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:56.771824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:56.773129 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.773543 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.773544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.773544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.773604 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.773863 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.773864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.773865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.773977 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:56.774052 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.774318 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.774393 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:50:56.774662 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.766066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41705 (local address 127.5.15.131:59115)
0926 14:50:56.766230 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.766261 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.766727 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.766983 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.766985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.767496 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:50:56.767665 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.767670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.768591 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.768593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.769569 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.769572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.769903 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.769903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.770000 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.770372 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:50:56.770380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:50:56.771687 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:50:56.773227 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:56.773233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:56.773235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:56.773450 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:56.773673 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:50:56.773674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:50:56.773675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:50:56.773789 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:50:56.774044 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:50:56.774050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:50:56.774184 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.774296 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.774442 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260926 14:50:56.775187 15860 heartbeater.cc:347] Connected to a master server at 127.5.15.190:41705
I20260926 14:50:56.775260 15860 heartbeater.cc:464] Registering TS with master...
I20260926 14:50:56.775451 15860 heartbeater.cc:511] Master 127.5.15.190:41705 requested a full tablet report, sending...
I20260926 14:50:56.775803 15401 ts_manager.cc:194] Registered new tserver with Master: 0bbef3af502c43ebb11a1de4c743989f (127.5.15.131:44241)
I20260926 14:50:56.776170 15401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:59115
I20260926 14:50:56.779884 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:50:56.786401 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.781084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.5.15.190:41705)
0926 14:50:56.781136 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.781138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.781246 (+ 108us) server_negotiation.cc:407] Connection header received
0926 14:50:56.781292 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.781292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.781300 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.781347 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:50:56.781829 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.782249 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.783005 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.783173 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.783222 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.784306 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:50:56.784307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:50:56.784307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:50:56.784313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:50:56.785409 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.785725 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.785725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.785726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.785770 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:50:56.785961 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:50:56.785961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:50:56.785962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:50:56.786081 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:50:56.786152 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.786253 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.786307 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:50:56.788020 15401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:50:56.789526 15401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:50:56.795720 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.791724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.5.15.130:33403)
0926 14:50:56.791926 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.791931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.792235 (+ 304us) server_negotiation.cc:407] Connection header received
0926 14:50:56.792298 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.792300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.792316 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.792397 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:56.792581 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.793330 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.794646 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.795344 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.795486 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.795601 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.795641 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260926 14:50:56.795725 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.791627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:33403 (local address 127.0.0.1:34512)
0926 14:50:56.792151 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.792161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.792227 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.792415 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.792418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.792420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:56.792481 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.792483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.793424 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.793424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.794565 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.794567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.795428 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.795504 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.795516 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.795588 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.795639 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":70,"threads_started":1}
I20260926 14:50:56.796123 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.791598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:38911 (local address 127.0.0.1:53336)
0926 14:50:56.791906 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.791964 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.792062 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.792447 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.792448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.792450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:56.792507 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.792509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.793801 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.793802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.794878 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.794880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.795829 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.795908 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.795912 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.795961 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.796023 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":55,"threads_started":1}
I20260926 14:50:56.796142 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.791692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.5.15.129:38911)
0926 14:50:56.792062 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.792065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.792077 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:50:56.792179 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.792181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.792197 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.792281 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:56.792591 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.793692 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.794955 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.795665 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795738 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.795791 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.796001 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.796068 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":99,"threads_started":1}
I20260926 14:50:56.796123 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.791782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:44241 (local address 127.0.0.1:39286)
0926 14:50:56.792305 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.792314 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.792363 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.792682 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.792683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.792685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:56.792724 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.792725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.793953 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.793954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.795026 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.795028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.795901 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.795957 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.795961 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.796016 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.796061 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":77,"threads_started":1}
I20260926 14:50:56.796748 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.791874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.5.15.131:44241)
0926 14:50:56.792072 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.792076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.792494 (+ 418us) server_negotiation.cc:407] Connection header received
0926 14:50:56.792532 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.792534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.792550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.792618 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:56.792812 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.793850 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795111 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.795768 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.795828 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.795873 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.796162 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.796664 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":102,"threads_started":1}
I20260926 14:50:56.796867 15660 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4f4af39d654bf78820c015f2781c58 (DEFAULT_TABLE table=test-table [id=526b7ed83fd64d18b2646e91cc101136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:56.796991 15526 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4f4af39d654bf78820c015f2781c58 (DEFAULT_TABLE table=test-table [id=526b7ed83fd64d18b2646e91cc101136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:56.797214 15660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4f4af39d654bf78820c015f2781c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:56.797258 15526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4f4af39d654bf78820c015f2781c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:56.800532 15877 tablet_bootstrap.cc:492] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Bootstrap starting.
I20260926 14:50:56.800635 15794 tablet_service.cc:1467] Processing CreateTablet for tablet 2c4f4af39d654bf78820c015f2781c58 (DEFAULT_TABLE table=test-table [id=526b7ed83fd64d18b2646e91cc101136]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:50:56.800743 15878 tablet_bootstrap.cc:492] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e: Bootstrap starting.
I20260926 14:50:56.800954 15794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c4f4af39d654bf78820c015f2781c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:50:56.801378 15878 tablet_bootstrap.cc:654] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:56.801941 15878 log.cc:824] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:56.802775 15877 tablet_bootstrap.cc:654] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:56.803323 15877 log.cc:824] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:56.803881 15878 tablet_bootstrap.cc:492] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e: No bootstrap required, opened a new log
I20260926 14:50:56.803956 15878 ts_tablet_manager.cc:1397] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:50:56.804039 15877 tablet_bootstrap.cc:492] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: No bootstrap required, opened a new log
I20260926 14:50:56.804100 15877 ts_tablet_manager.cc:1397] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:50:56.804368 15881 tablet_bootstrap.cc:492] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Bootstrap starting.
I20260926 14:50:56.805047 15881 tablet_bootstrap.cc:654] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Neither blocks nor log segments found. Creating new log.
I20260926 14:50:56.805559 15881 log.cc:824] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:50:56.805603 15878 raft_consensus.cc:348] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.805815 15878 raft_consensus.cc:374] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:56.805891 15878 raft_consensus.cc:729] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17d2d371a743c88e7ec4dd3c40dd4e, State: Initialized, Role: FOLLOWER
I20260926 14:50:56.806041 15878 consensus_queue.cc:260] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.806324 15878 ts_tablet_manager.cc:1428] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:56.806427 15881 tablet_bootstrap.cc:492] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: No bootstrap required, opened a new log
I20260926 14:50:56.806316 15877 raft_consensus.cc:348] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.806473 15877 raft_consensus.cc:374] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:56.806494 15881 ts_tablet_manager.cc:1397] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:50:56.806495 15592 heartbeater.cc:503] Master 127.5.15.190:41705 was elected leader, sending a full tablet report...
I20260926 14:50:56.806505 15877 raft_consensus.cc:729] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfeb5fdda1494a4cbe2b9d3d812a3a57, State: Initialized, Role: FOLLOWER
I20260926 14:50:56.806648 15877 consensus_queue.cc:260] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.806928 15877 ts_tablet_manager.cc:1428] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:50:56.806977 15726 heartbeater.cc:503] Master 127.5.15.190:41705 was elected leader, sending a full tablet report...
I20260926 14:50:56.808367 15881 raft_consensus.cc:348] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.808517 15881 raft_consensus.cc:374] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:50:56.808558 15881 raft_consensus.cc:729] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbef3af502c43ebb11a1de4c743989f, State: Initialized, Role: FOLLOWER
I20260926 14:50:56.808656 15881 consensus_queue.cc:260] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.808895 15881 ts_tablet_manager.cc:1428] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 14:50:56.808946 15860 heartbeater.cc:503] Master 127.5.15.190:41705 was elected leader, sending a full tablet report...
I20260926 14:50:56.814987 15883 raft_consensus.cc:482] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:50:56.815052 15883 raft_consensus.cc:504] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.815331 15883 leader_election.cc:288] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bbef3af502c43ebb11a1de4c743989f (127.5.15.131:44241), cfeb5fdda1494a4cbe2b9d3d812a3a57 (127.5.15.130:33403)
I20260926 14:50:56.818035 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.815486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:37317 (local address 127.5.15.131:44241)
0926 14:50:56.815521 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.815523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.815590 (+ 67us) server_negotiation.cc:407] Connection header received
0926 14:50:56.815613 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.815614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.815623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.815705 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:56.815830 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.816385 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.817285 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.817704 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.817756 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.817764 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.817930 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.817970 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:50:56.818034 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.815466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:44241 (local address 127.5.15.129:37317)
0926 14:50:56.815510 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.815521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.815566 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.815707 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.815708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.815710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:56.815770 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.815772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.816457 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.816457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.817194 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.817196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.817777 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.817778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.817858 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.817863 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.817923 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.817952 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 14:50:56.818128 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.815532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:53681 (local address 127.5.15.130:33403)
0926 14:50:56.815578 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.815581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.815688 (+ 107us) server_negotiation.cc:407] Connection header received
0926 14:50:56.815802 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.815803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.815813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.815842 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:50:56.816026 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.816487 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.817402 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.817787 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.817866 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.817874 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.818021 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.818060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:50:56.818157 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.815466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:33403 (local address 127.5.15.129:53681)
0926 14:50:56.815634 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.815643 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:50:56.815713 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:50:56.815901 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:50:56.815901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:50:56.815903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:50:56.815960 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.815962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.816562 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.816562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.817320 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:50:56.817322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.817864 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.817864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:50:56.817944 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.817949 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:50:56.818004 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:50:56.818039 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":11,"thread_start_us":78,"threads_started":1}
I20260926 14:50:56.818351 15814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f4af39d654bf78820c015f2781c58" candidate_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbef3af502c43ebb11a1de4c743989f" is_pre_election: true
I20260926 14:50:56.818466 15814 raft_consensus.cc:2454] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a17d2d371a743c88e7ec4dd3c40dd4e in term 0.
I20260926 14:50:56.818449 15680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f4af39d654bf78820c015f2781c58" candidate_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" is_pre_election: true
I20260926 14:50:56.818581 15680 raft_consensus.cc:2454] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a17d2d371a743c88e7ec4dd3c40dd4e in term 0.
I20260926 14:50:56.818693 15482 leader_election.cc:302] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bbef3af502c43ebb11a1de4c743989f, 4a17d2d371a743c88e7ec4dd3c40dd4e; no voters:
I20260926 14:50:56.818864 15883 raft_consensus.cc:2781] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:50:56.818925 15883 raft_consensus.cc:482] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:50:56.818960 15883 raft_consensus.cc:3037] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:56.819574 15883 raft_consensus.cc:504] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.819712 15883 leader_election.cc:288] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [CANDIDATE]: Term 1 election: Requested vote from peers 0bbef3af502c43ebb11a1de4c743989f (127.5.15.131:44241), cfeb5fdda1494a4cbe2b9d3d812a3a57 (127.5.15.130:33403)
I20260926 14:50:56.819851 15680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f4af39d654bf78820c015f2781c58" candidate_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57"
I20260926 14:50:56.819921 15680 raft_consensus.cc:3037] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:56.820542 15680 raft_consensus.cc:2454] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a17d2d371a743c88e7ec4dd3c40dd4e in term 1.
I20260926 14:50:56.820668 15814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c4f4af39d654bf78820c015f2781c58" candidate_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bbef3af502c43ebb11a1de4c743989f"
I20260926 14:50:56.820703 15479 leader_election.cc:302] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17d2d371a743c88e7ec4dd3c40dd4e, cfeb5fdda1494a4cbe2b9d3d812a3a57; no voters:
I20260926 14:50:56.820735 15814 raft_consensus.cc:3037] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:50:56.820801 15883 raft_consensus.cc:2781] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:50:56.820953 15883 raft_consensus.cc:686] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [term 1 LEADER]: Becoming Leader. State: Replica: 4a17d2d371a743c88e7ec4dd3c40dd4e, State: Running, Role: LEADER
I20260926 14:50:56.821025 15883 consensus_queue.cc:237] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } }
I20260926 14:50:56.821552 15814 raft_consensus.cc:2454] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a17d2d371a743c88e7ec4dd3c40dd4e in term 1.
I20260926 14:50:56.821703 15401 catalog_manager.cc:5495] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a17d2d371a743c88e7ec4dd3c40dd4e (127.5.15.129). New cstate: current_term: 1 leader_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17d2d371a743c88e7ec4dd3c40dd4e" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 } health_report { overall_health: UNKNOWN } } }
I20260926 14:50:56.833868 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.831610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.5.15.129:38911)
0926 14:50:56.831658 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 14:50:56.831660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:50:56.831669 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:50:56.831735 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:50:56.831735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:50:56.831755 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:50:56.831841 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:50:56.831946 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.832377 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.833111 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:50:56.833294 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:50:56.833365 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:50:56.833537 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:50:56.833622 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:50:56.833706 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:50:56.833750 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 14:50:56.833767 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 14:50:56.835470 15814 raft_consensus.cc:1261] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 1 FOLLOWER]: Refusing update from remote peer 4a17d2d371a743c88e7ec4dd3c40dd4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:56.835474 15680 raft_consensus.cc:1261] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 1 FOLLOWER]: Refusing update from remote peer 4a17d2d371a743c88e7ec4dd3c40dd4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:50:56.835810 15883 consensus_queue.cc:1035] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfeb5fdda1494a4cbe2b9d3d812a3a57" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:56.835956 15887 consensus_queue.cc:1035] T 2c4f4af39d654bf78820c015f2781c58 P 4a17d2d371a743c88e7ec4dd3c40dd4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbef3af502c43ebb11a1de4c743989f" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:50:56.838820 15890 mvcc.cc:205] Tried to move back new op lower bound from 7333618715995672576 to 7333618715939401728. Current Snapshot: MvccSnapshot[applied={T|T < 7333618715995672576 or (T in {7333618715995672576})}]
I20260926 14:50:56.839380 15891 mvcc.cc:205] Tried to move back new op lower bound from 7333618715995672576 to 7333618715939401728. Current Snapshot: MvccSnapshot[applied={T|T < 7333618715995672576}]
I20260926 14:50:56.840658 15400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:56.840747 15400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:50:56.841472 15400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64547eb69bad4df08f4b3dacf46fec2e: Sending DeleteTablet for 3 replicas of tablet 2c4f4af39d654bf78820c015f2781c58
I20260926 14:50:56.841797 15660 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4f4af39d654bf78820c015f2781c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:56 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:34512
I20260926 14:50:56.842037 15899 tablet_replica.cc:330] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: stopping tablet replica
I20260926 14:50:56.842147 15899 raft_consensus.cc:2229] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:56.842125 15794 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4f4af39d654bf78820c015f2781c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:56 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:39286
I20260926 14:50:56.842275 15899 raft_consensus.cc:2258] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:56.842350 15900 tablet_replica.cc:330] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: stopping tablet replica
I20260926 14:50:56.842411 15526 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c4f4af39d654bf78820c015f2781c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:50:56 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:53336
I20260926 14:50:56.842479 15900 raft_consensus.cc:2229] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:50:56.842576 15900 raft_consensus.cc:2258] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:50:56.842947 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 15463
I20260926 14:50:56.843035 15900 ts_tablet_manager.cc:1905] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:56.842986 15899 ts_tablet_manager.cc:1905] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:50:56.845465 15899 ts_tablet_manager.cc:1918] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:56.845501 15900 ts_tablet_manager.cc:1918] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:50:56.845567 15900 log.cc:1196] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/2c4f4af39d654bf78820c015f2781c58
I20260926 14:50:56.845570 15899 log.cc:1196] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/2c4f4af39d654bf78820c015f2781c58
I20260926 14:50:56.845842 15899 ts_tablet_manager.cc:1939] T 2c4f4af39d654bf78820c015f2781c58 P cfeb5fdda1494a4cbe2b9d3d812a3a57: Deleting consensus metadata
I20260926 14:50:56.845881 15900 ts_tablet_manager.cc:1939] T 2c4f4af39d654bf78820c015f2781c58 P 0bbef3af502c43ebb11a1de4c743989f: Deleting consensus metadata
I20260926 14:50:56.846262 15385 catalog_manager.cc:4841] TS cfeb5fdda1494a4cbe2b9d3d812a3a57 (127.5.15.130:33403): tablet 2c4f4af39d654bf78820c015f2781c58 (table test-table [id=526b7ed83fd64d18b2646e91cc101136]) successfully deleted
I20260926 14:50:56.846349 15388 catalog_manager.cc:4841] TS 0bbef3af502c43ebb11a1de4c743989f (127.5.15.131:44241): tablet 2c4f4af39d654bf78820c015f2781c58 (table test-table [id=526b7ed83fd64d18b2646e91cc101136]) successfully deleted
W20260926 14:50:56.848611 15388 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:38911 (error 108)
I20260926 14:50:56.848783 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 15597
I20260926 14:50:56.849354 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:50:56.848969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:38911 (local address 127.0.0.1:53358)
0926 14:50:56.849286 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 14:50:56.849327 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:38911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
W20260926 14:50:56.849519 15388 catalog_manager.cc:4584] TS 4a17d2d371a743c88e7ec4dd3c40dd4e (127.5.15.129:38911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c4f4af39d654bf78820c015f2781c58: Network error: Client connection negotiation failed: client connection to 127.5.15.129:38911: connect: Connection refused (error 111)
I20260926 14:50:56.853412 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 15731
I20260926 14:50:56.857651 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 15369
2026-09-26T14:50:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1886 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15905](info): set up 2 sockets
Sep 26 14:50:56 dist-test-slave-s4p1 krb5kdc[15905](info): commencing operation
krb5kdc: starting...
W20260926 14:50:57.883790 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:50:57 dist-test-slave-s4p1 krb5kdc[15905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:50:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/rangeradmin_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/rangeradmin_127.5.15.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/rangerlookup_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/rangerlookup_127.5.15.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/HTTP_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/krb5kdc/HTTP_127.5.15.148@KRBTEST.COM.keytab.
I20260926 14:50:58.036705 5182 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmCUWsf/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/postgres -l logfile start
2026-09-26 14:51:00.359 UTC [15935] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:51:00.359 UTC [15935] LOG: listening on IPv4 address "127.5.15.148", port 34991
2026-09-26 14:51:00.362 UTC [15935] LOG: listening on Unix socket "/tmp/.s.PGSQL.34991"
2026-09-26 14:51:00.396 UTC [15938] LOG: database system was shut down at 2026-09-26 14:50:59 UTC
2026-09-26 14:51:00.400 UTC [15935] LOG: database system is ready to accept connections
I20260926 14:51:01.335973 5182 mini_postgres.cc:96] Postgres bound to 34991
2026-09-26 14:51:01.341 UTC [15946] FATAL: database "slave" does not exist
127.5.15.148:34991 - accepting connections
I20260926 14:51:01.342465 5182 mini_ranger.cc:161] Starting Ranger...
I20260926 14:51:01.356084 5182 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:51:01.790827 5182 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:51:01.790889 5182 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/ranger-admin
2026-09-26 14:51:02,056 [I] DB FLAVOR :POSTGRES
2026-09-26 14:51:02,057 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:51:02,057 [I] Checking connection..
2026-09-26 14:51:02,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:51:02,301 [I] Checking connection passed.
2026-09-26 14:51:02,301 [I] --------- Verifying version history table ---------
2026-09-26 14:51:02,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:02,550 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:51:02,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:02,801 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:51:02,801 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:51:02,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:51:03.024 UTC [16010] WARNING: there is no transaction in progress
2026-09-26 14:51:03,030 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:51:03,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:51:03,278 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:51:03,278 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:51:03,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:51:03,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:51:03,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s4p1.c.gcp-upstream.internal','N') ;"
2026-09-26 14:51:03,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:51:04,189 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:51:04,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:51:04,435 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:51:04,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:51:04,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:51:04,932 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:51:04,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:51:05.847 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.864 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.880 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.895 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.910 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.927 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.980 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:05.987 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.001 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.011 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.020 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.033 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.042 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.048 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.054 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06.060 UTC [16118] WARNING: there is no transaction in progress
2026-09-26 14:51:06,642 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:51:06,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:51:06,890 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:51:06,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:51:07,143 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:51:07,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:51:07,379 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:51:07,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:51:07,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:51:07,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s4p1.c.gcp-upstream.internal';"
2026-09-26 14:51:08,094 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:51:08,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:34991/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:51:08,339 [I] DB_PATCHES have already been applied
I20260926 14:51:08.344457 5182 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmCUWsf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:51:08.344519 5182 mini_ranger.cc:193] Using host: 127.5.15.148
I20260926 14:51:08.347481 5182 mini_ranger.cc:227] Ranger admin URL: http://127.5.15.148:39523
Sep 26, 2026 2:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:51:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:51:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 26, 2026 2:51:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:51:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 2:51:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:51:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:51:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:53:09.218616 5182 mini_ranger.cc:160] Time spent starting Ranger: real 127.876s user 0.000s sys 0.140s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 16205 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p16205
f310
n127.0.0.1:60868->127.5.15.148:34991
f311
n127.0.0.1:60878->127.5.15.148:34991
f312
n127.0.0.1:60890->127.5.15.148:34991
f313
n127.0.0.1:60902->127.5.15.148:34991
f314
n127.0.0.1:60918->127.5.15.148:34991'
I20260926 14:53:09.219444 5182 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:53:09.240113 5182 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:53:09.240 UTC [15935] LOG: received smart shutdown request
2026-09-26 14:53:09.243 UTC [15935] LOG: background worker "logical replication launcher" (PID 15944) exited with exit code 1
2026-09-26 14:53:09.244 UTC [15939] LOG: shutting down
2026-09-26 14:53:09.412 UTC [15935] LOG: database system is shut down
2026-09-26T14:53:09Z chronyd exiting
I20260926 14:53:09.433830 5182 test_util.cc:175] -----------------------------------------------
I20260926 14:53:09.433889 5182 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434201368912-5182-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (132556 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:09 dist-test-slave-s4p1 krb5kdc[16327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:09 dist-test-slave-s4p1 krb5kdc[16327](info): set up 2 sockets
Sep 26 14:53:09 dist-test-slave-s4p1 krb5kdc[16327](info): commencing operation
krb5kdc: starting...
W20260926 14:53:10.454101 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:53:10 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/rangeradmin_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/rangeradmin_127.5.15.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/rangerlookup_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/rangerlookup_127.5.15.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/HTTP_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/HTTP_127.5.15.148@KRBTEST.COM.keytab.
I20260926 14:53:10.602361 5182 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmCUWsf/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/postgres -l logfile start
2026-09-26 14:53:12.892 UTC [16357] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:53:12.892 UTC [16357] LOG: listening on IPv4 address "127.5.15.148", port 50917
2026-09-26 14:53:12.895 UTC [16357] LOG: listening on Unix socket "/tmp/.s.PGSQL.50917"
2026-09-26 14:53:12.915 UTC [16360] LOG: database system was shut down at 2026-09-26 14:53:11 UTC
2026-09-26 14:53:12.918 UTC [16357] LOG: database system is ready to accept connections
I20260926 14:53:13.862044 5182 mini_postgres.cc:96] Postgres bound to 50917
2026-09-26 14:53:13.867 UTC [16368] FATAL: database "slave" does not exist
127.5.15.148:50917 - accepting connections
I20260926 14:53:13.868394 5182 mini_ranger.cc:161] Starting Ranger...
I20260926 14:53:13.882454 5182 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:53:14.329563 5182 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:53:14.329645 5182 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-admin
2026-09-26 14:53:14,601 [I] DB FLAVOR :POSTGRES
2026-09-26 14:53:14,602 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:53:14,602 [I] Checking connection..
2026-09-26 14:53:14,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:53:14,828 [I] Checking connection passed.
2026-09-26 14:53:14,828 [I] --------- Verifying version history table ---------
2026-09-26 14:53:14,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:53:15,082 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:53:15,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:53:15,317 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:53:15,317 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:53:15,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:53:15.541 UTC [16432] WARNING: there is no transaction in progress
2026-09-26 14:53:15,547 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:53:15,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:53:15,775 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:53:15,776 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:53:15,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:53:16,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:53:16,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s4p1.c.gcp-upstream.internal','N') ;"
2026-09-26 14:53:16,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:53:16,733 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:53:16,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:53:16,986 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:53:16,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:53:17,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:53:17,448 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:53:17,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:53:18.377 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.392 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.409 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.425 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.443 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.463 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.527 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.533 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.546 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.556 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.565 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.578 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.587 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.593 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.600 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:18.606 UTC [16540] WARNING: there is no transaction in progress
2026-09-26 14:53:19,219 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:53:19,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:53:19,465 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:53:19,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:53:19,725 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:53:19,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:53:19,971 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:53:19,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:53:20,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:53:20,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s4p1.c.gcp-upstream.internal';"
2026-09-26 14:53:20,698 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:53:20,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:53:20,932 [I] DB_PATCHES have already been applied
I20260926 14:53:20.937469 5182 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmCUWsf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:53:20.937551 5182 mini_ranger.cc:193] Using host: 127.5.15.148
I20260926 14:53:20.940510 5182 mini_ranger.cc:227] Ranger admin URL: http://127.5.15.148:36609
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:53:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:53:22.027318 5182 mini_ranger.cc:160] Time spent starting Ranger: real 8.159s user 0.000s sys 0.009s
Sep 26, 2026 2:53:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:53:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:53:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:53:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:53:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:53:42.112470 5182 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.15.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.15.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/rangerkms_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.15.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/rangerkms_127.5.15.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.15.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.15.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.15.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/HTTP_127.5.15.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.15.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/HTTP_127.5.15.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:53:42 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:53:42.203389 5182 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:53:42.212625 5182 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:53:42.999547 5182 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:53:42.999611 5182 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms
I20260926 14:53:42.999624 5182 mini_ranger_kms.cc:227] Using postgres at 127.5.15.148:50917
2026-09-26 14:53:43,160 [I] DB FLAVOR :POSTGRES
2026-09-26 14:53:43,160 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:53:43,161 [I] Checking connection
2026-09-26 14:53:43,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:53:43,382 [I] connection success
2026-09-26 14:53:43,383 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:53:43,383 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:53:43,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:53:43,629 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:53:43,629 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:53:43,629 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:53:43,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.15.148:50917/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:53:43,896 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:53:44.345615 5182 mini_ranger_kms.cc:310] Created kms service
I20260926 14:53:44.545908 5182 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:53:44.588758 5182 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:53:44 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:44.966724 5182 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:53:44.966879 5182 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmCUWsf/build/release/bin/postgresql.jar:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmCUWsf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:53:44.966903 5182 mini_ranger_kms.cc:240] Using host: 127.5.15.148
I20260926 14:53:44.970110 5182 mini_ranger_kms.cc:276] Ranger KMS PID: 16720
I20260926 14:53:44.970225 5182 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.15.148:42529
Sep 26, 2026 2:53:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:53:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:53:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:53:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:53:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 14:53:47 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.15.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:47.130936 5182 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.928s user 0.000s sys 0.145s
I20260926 14:53:47.131057 5182 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:53:47.131166 5182 mini_ranger_kms.cc:401] 127.5.15.148:42529/kms/v1/keys
Sep 26 14:53:47 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.15.148@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
Sep 26 14:53:47 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.15.148@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
Sep 26 14:53:49 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:53:49.792672 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36117
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36117
--ranger_config_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:49.870148 16771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:49.870326 16771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:49.870347 16771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:49.871539 16771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:49.871580 16771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:49.871594 16771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:49.871608 16771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:49.871657 16771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:49.873095 16771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.15.148:42529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36117
--ranger_config_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:49.873320 16771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:49.873528 16771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:49.878346 16779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:49.878338 16776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:49.878469 16777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:49.878399 16771 server_base.cc:1034] running on GCE node
I20260926 14:53:49.878823 16771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:49.879108 16771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:49.880275 16771 hybrid_clock.cc:648] HybridClock initialized: now 1790434429880261 us; error 44 us; skew 500 ppm
Sep 26 14:53:49 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434429, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:49.889201 16771 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:53:49.889626 16771 webserver.cc:460] Webserver started at http://127.5.15.190:36239/ using document root <none> and password file <none>
I20260926 14:53:49.889812 16771 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:49.889858 16771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:49.889941 16771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:53:49 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434429, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:49.933269 16771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "faacac3ef86d4aca95312d01f061ce83"
format_stamp: "Formatted at 2026-09-26 14:53:49 on dist-test-slave-s4p1"
server_key: "1a80ab83281044070af34c3c8e3e9c74"
server_key_iv: "f1eb03c5b42d842cb54d61a8c284fbdb"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:49.933656 16771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "faacac3ef86d4aca95312d01f061ce83"
format_stamp: "Formatted at 2026-09-26 14:53:49 on dist-test-slave-s4p1"
server_key: "1a80ab83281044070af34c3c8e3e9c74"
server_key_iv: "f1eb03c5b42d842cb54d61a8c284fbdb"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:49.935066 16771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:53:49.945611 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:49.945822 16771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:53:49.945915 16771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "faacac3ef86d4aca95312d01f061ce83"
format_stamp: "Formatted at 2026-09-26 14:53:49 on dist-test-slave-s4p1"
server_key: "1a80ab83281044070af34c3c8e3e9c74"
server_key_iv: "f1eb03c5b42d842cb54d61a8c284fbdb"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:49.945995 16771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:49.960197 16771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:49.962222 16771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:49.962357 16771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:49.966444 16771 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:36117
I20260926 14:53:49.966485 16838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:36117 every 8 connection(s)
I20260926 14:53:49.966879 16771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:49.967542 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 16771
I20260926 14:53:49.967553 16839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:49.967664 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:53:49.967875 5182 external_mini_cluster.cc:1428] Setting key 30aa81a9023a6e2d20d96616a414b65e
I20260926 14:53:49.970976 16839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Bootstrap starting.
I20260926 14:53:49.971661 16839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:49.971962 16839 log.cc:824] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:49.972751 16839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: No bootstrap required, opened a new log
Sep 26 14:53:49 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:49.975317 16839 raft_consensus.cc:348] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:49.975476 16839 raft_consensus.cc:374] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:49.975539 16839 raft_consensus.cc:729] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faacac3ef86d4aca95312d01f061ce83, State: Initialized, Role: FOLLOWER
I20260926 14:53:49.975687 16839 consensus_queue.cc:260] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:49.975761 16839 raft_consensus.cc:388] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:49.975785 16839 raft_consensus.cc:482] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:49.975844 16839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:49.976372 16839 raft_consensus.cc:504] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:49.976488 16839 leader_election.cc:302] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faacac3ef86d4aca95312d01f061ce83; no voters:
I20260926 14:53:49.976670 16839 leader_election.cc:288] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:49.976725 16844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:49.976890 16844 raft_consensus.cc:686] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 LEADER]: Becoming Leader. State: Replica: faacac3ef86d4aca95312d01f061ce83, State: Running, Role: LEADER
I20260926 14:53:49.976917 16839 sys_catalog.cc:564] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:49.977028 16844 consensus_queue.cc:237] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:49.977365 16844 sys_catalog.cc:455] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faacac3ef86d4aca95312d01f061ce83. Latest consensus state: current_term: 1 leader_uuid: "faacac3ef86d4aca95312d01f061ce83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } } }
I20260926 14:53:49.977408 16845 sys_catalog.cc:455] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faacac3ef86d4aca95312d01f061ce83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } } }
I20260926 14:53:49.977519 16845 sys_catalog.cc:458] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:49.977766 16844 sys_catalog.cc:458] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:49.978195 16852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:49.978530 16839 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:53:49.978734 16852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:49.979526 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:49.968439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.5.15.190:36117)
0926 14:53:49.968645 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:49.968655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:49.968687 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:53:49.969214 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:49.969222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:49.969287 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:49.969437 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:49.970053 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:49.971455 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:49.972360 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:49.972886 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:49.972979 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:49.975573 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:49.975577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:49.975577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:49.975605 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:49.977508 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:49.977936 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:49.977954 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:49.977955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:49.978064 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:49.978286 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:49.978286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:49.978287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:49.978541 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:49.978703 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:49.979221 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 14:53:49.979332 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260926 14:53:49.983865 16852 catalog_manager.cc:1347] Generated new cluster ID: dc4ba89692494de59a6e85625cdea078
I20260926 14:53:49.983951 16852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:50.007553 16852 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:50.008275 16852 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:50.014304 16852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Generated new TSK 0
I20260926 14:53:50.014465 16852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:53:50.669831 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36117
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:50.781822 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:50.782076 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:50.782109 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:50.783794 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:50.783838 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:50.783865 16884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:53:50.785390 16884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.15.148:42529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:50.785557 16884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:50.785741 16884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:50.791477 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:50.791477 16890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:50.791698 16884 server_base.cc:1034] running on GCE node
W20260926 14:53:50.791749 16889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:50.791963 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:50.792200 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:50.793362 16884 hybrid_clock.cc:648] HybridClock initialized: now 1790434430793350 us; error 31 us; skew 500 ppm
Sep 26 14:53:50 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434430, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:50.796353 16884 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:53:50.796836 16884 webserver.cc:460] Webserver started at http://127.5.15.129:35511/ using document root <none> and password file <none>
I20260926 14:53:50.797066 16884 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:50.797125 16884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:50.797300 16884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:53:50 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434430, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:50.811729 16884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "d073945ab81949049b1a5167b1ffad47"
format_stamp: "Formatted at 2026-09-26 14:53:50 on dist-test-slave-s4p1"
server_key: "a4a5fa1fee84f10a9dcf8344dc3fdef7"
server_key_iv: "42888ba34c4b1112f1702d256b6dbb60"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:50.812168 16884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "d073945ab81949049b1a5167b1ffad47"
format_stamp: "Formatted at 2026-09-26 14:53:50 on dist-test-slave-s4p1"
server_key: "a4a5fa1fee84f10a9dcf8344dc3fdef7"
server_key_iv: "42888ba34c4b1112f1702d256b6dbb60"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:50.813911 16884 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:53:50.826882 16899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:50.827195 16884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:53:50.827349 16884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "d073945ab81949049b1a5167b1ffad47"
format_stamp: "Formatted at 2026-09-26 14:53:50 on dist-test-slave-s4p1"
server_key: "a4a5fa1fee84f10a9dcf8344dc3fdef7"
server_key_iv: "42888ba34c4b1112f1702d256b6dbb60"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:50.827499 16884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:50.840163 16884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:50.842639 16884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:50.842876 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:50.843140 16884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:50.843506 16884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:50.843587 16884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:50.843698 16884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:50.843755 16884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:50.850308 16884 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:42013
I20260926 14:53:50.850735 16884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:50.850797 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:42013 every 8 connection(s)
I20260926 14:53:50.855693 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 16884
I20260926 14:53:50.855783 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:53:50.855966 5182 external_mini_cluster.cc:1428] Setting key 8e8fd035c4aedb20b7e5a96ef615f4dd
Sep 26 14:53:50 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434430, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:50.862600 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:50.851771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36117 (local address 127.5.15.129:57853)
0926 14:53:50.851960 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:50.852008 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:50.852634 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:50.853097 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:50.853103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:50.853327 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:50.853588 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:50.853593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:50.854591 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:50.854593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:50.855975 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:50.855978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:50.856561 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:50.856562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:50.856712 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:50.857295 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:50.857306 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:50.858966 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:50.861064 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:50.861071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:50.861073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:50.861367 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:50.861584 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:50.861585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:50.861585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:50.861715 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:50.861997 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:50.862002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:50.862147 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 14:53:50.862232 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:53:50.862373 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260926 14:53:50.863375 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:50.852039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:57853 (local address 127.5.15.190:36117)
0926 14:53:50.852839 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:50.852841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:50.852850 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:53:50.852876 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:50.852876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:50.852890 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:50.852980 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:50.853715 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:50.854472 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:50.856082 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:50.856448 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:50.859058 (+ 2610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:50.859111 (+ 53us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:50.859112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:50.859112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:50.859117 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:50.860965 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:50.861455 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:50.861456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:50.861457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:50.861516 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:50.861794 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:50.861794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:50.861795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:50.861925 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:50.863152 (+ 1227us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:50.863195 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 14:53:50.863276 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":70,"threads_started":1}
I20260926 14:53:50.864023 17015 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36117
I20260926 14:53:50.864120 17015 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:50.864316 17015 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:50.864915 16803 ts_manager.cc:194] Registered new tserver with Master: d073945ab81949049b1a5167b1ffad47 (127.5.15.129:42013)
I20260926 14:53:50.865566 16803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:57853
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:53:50.891640 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36117
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:51.000552 17021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:51.000873 17021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:51.000941 17021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:51.002658 17021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:51.002779 17021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:51.002821 17021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:53:51.004985 17021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.15.148:42529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:51.005260 17021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:51.005541 17021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 14:53:51.011739 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.011793 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.011929 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.012655 17021 server_base.cc:1034] running on GCE node
I20260926 14:53:51.012876 17021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:51.013124 17021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:51.014307 17021 hybrid_clock.cc:648] HybridClock initialized: now 1790434431014284 us; error 34 us; skew 500 ppm
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:51.017659 17021 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:53:51.018182 17021 webserver.cc:460] Webserver started at http://127.5.15.130:34463/ using document root <none> and password file <none>
I20260926 14:53:51.018478 17021 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:51.018580 17021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:51.018800 17021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:51.032907 17021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "22ece3465b6744ebafd451ad7540df73"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "189b71d416f6a23499cd28d10a3b7737"
server_key_iv: "4fe7505a54e979dca7df523c94411388"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.033365 17021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "22ece3465b6744ebafd451ad7540df73"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "189b71d416f6a23499cd28d10a3b7737"
server_key_iv: "4fe7505a54e979dca7df523c94411388"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.040740 17021 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.002s
I20260926 14:53:51.061311 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.061815 17021 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 14:53:51.061932 17021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "22ece3465b6744ebafd451ad7540df73"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "189b71d416f6a23499cd28d10a3b7737"
server_key_iv: "4fe7505a54e979dca7df523c94411388"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.062031 17021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:51.084244 17021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:51.086651 17021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:51.086889 17021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:51.087198 17021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:51.087587 17021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:51.087667 17021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.087749 17021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:51.087805 17021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.094120 17021 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:39215
I20260926 14:53:51.094584 17021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:53:51.096953 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17021
I20260926 14:53:51.097055 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:53:51.097282 5182 external_mini_cluster.cc:1428] Setting key 32b15bfe3cdc881eb3e702fb20115d1d
I20260926 14:53:51.097376 17152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:39215 every 8 connection(s)
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:51.109409 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.099736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:41775 (local address 127.5.15.190:36117)
0926 14:53:51.099781 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:51.099783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:51.099792 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:53:51.099814 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:51.099814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:51.099826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:51.099878 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:51.100737 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.101517 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.103177 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.103495 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.103541 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.106144 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:51.106145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:51.106145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:51.106152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:51.107888 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.108347 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.108348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.108348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.108402 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.108659 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.108660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.108661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.108816 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:51.108862 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:51.109212 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 14:53:51.109312 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:53:51.110107 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.098452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36117 (local address 127.5.15.130:41775)
0926 14:53:51.098832 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.098875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:51.099458 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:51.100085 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:51.100090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:51.100329 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:51.100612 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.100618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.101769 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.101770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.103077 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.103080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.103642 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.103643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.103790 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.104385 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:51.104395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:51.106035 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:51.107982 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.107988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.107990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.108258 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.108470 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.108470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.108471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.108588 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.108911 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:51.108916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:51.109064 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 14:53:51.109735 (+ 671us) client_negotiation.cc:240] Negotiation successful
0926 14:53:51.109888 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":86,"threads_started":1}
I20260926 14:53:51.110795 17153 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36117
I20260926 14:53:51.110885 17153 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.111054 17153 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:51.111474 16803 ts_manager.cc:194] Registered new tserver with Master: 22ece3465b6744ebafd451ad7540df73 (127.5.15.130:39215)
I20260926 14:53:51.111958 16803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:41775
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:53:51.138242 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36117
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:51.256145 17158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:51.256485 17158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:51.256580 17158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:51.258354 17158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:51.258414 17158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:51.258452 17158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:53:51.260591 17158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.15.148:42529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:51.261070 17158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:51.261333 17158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:51.267861 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.267944 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.268265 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.268697 17158 server_base.cc:1034] running on GCE node
I20260926 14:53:51.268927 17158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:51.269187 17158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:51.270360 17158 hybrid_clock.cc:648] HybridClock initialized: now 1790434431270334 us; error 41 us; skew 500 ppm
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:51.273600 17158 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:53:51.274098 17158 webserver.cc:460] Webserver started at http://127.5.15.131:45459/ using document root <none> and password file <none>
I20260926 14:53:51.274362 17158 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:51.274426 17158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:51.274559 17158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:51.289196 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "d11088afa6b147a098190de516cef61e"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "5c1ee730b8f02d4d09d94e61f04bf63a"
server_key_iv: "94dea809ac48cea9429851129ac30548"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.289597 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "d11088afa6b147a098190de516cef61e"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "5c1ee730b8f02d4d09d94e61f04bf63a"
server_key_iv: "94dea809ac48cea9429851129ac30548"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.291038 17158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:53:51.305480 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.305794 17158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:53:51.305928 17158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "d11088afa6b147a098190de516cef61e"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "5c1ee730b8f02d4d09d94e61f04bf63a"
server_key_iv: "94dea809ac48cea9429851129ac30548"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.306037 17158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:51.343308 17158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:51.346392 17158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:51.346606 17158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:51.346938 17158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:51.347383 17158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:51.347491 17158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.347572 17158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:51.347640 17158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.356148 17158 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46467
I20260926 14:53:51.356642 17158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:51.357290 17286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46467 every 8 connection(s)
I20260926 14:53:51.359542 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17158
I20260926 14:53:51.359633 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:53:51.359814 5182 external_mini_cluster.cc:1428] Setting key 7634cd1a92da076723f3644bda61dc10
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:51.368254 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.357456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:43885 (local address 127.5.15.190:36117)
0926 14:53:51.357512 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:51.357514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:51.357790 (+ 276us) server_negotiation.cc:407] Connection header received
0926 14:53:51.358350 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:51.358351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:51.358362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:51.358418 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:51.359304 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.360097 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.361786 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.362146 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.362194 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.364980 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:51.364981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:51.364982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:51.364990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:51.366791 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.367268 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.367269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.367270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.367329 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.367613 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.367613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.367614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.367746 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:51.367790 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:51.368091 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 14:53:51.368165 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:53:51.368455 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.357488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36117 (local address 127.5.15.131:43885)
0926 14:53:51.357655 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.357703 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:51.358240 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:51.358548 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:51.358553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:51.358852 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:51.359183 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.359190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.360292 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.360292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.361665 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.361668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.362304 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.362305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.362474 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.363153 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:51.363164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:51.364861 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:51.366884 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.366890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.366892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.367178 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.367398 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.367398 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.367399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.367529 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.367837 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:51.367842 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:51.367951 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 14:53:51.368070 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 14:53:51.368212 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260926 14:53:51.369036 17287 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36117
I20260926 14:53:51.369153 17287 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.369343 17287 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:51.369750 16803 ts_manager.cc:194] Registered new tserver with Master: d11088afa6b147a098190de516cef61e (127.5.15.131:46467)
I20260926 14:53:51.370136 16803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:43885
I20260926 14:53:51.371366 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:51.371975 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 16884
I20260926 14:53:51.378262 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17021
I20260926 14:53:51.384419 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17158
I20260926 14:53:51.389931 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 16771
I20260926 14:53:51.394333 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36117
--webserver_interface=127.5.15.190
--webserver_port=36239
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36117
--ranger_config_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:51.473716 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:51.473949 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:51.473985 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:51.475193 17290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:51.475252 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:51.475278 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:51.475322 17290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:51.475353 17290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:51.476883 17290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36117
--ranger_config_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=36239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:51.477164 17290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:51.477382 17290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:51.482306 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.482333 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.482333 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.482522 17290 server_base.cc:1034] running on GCE node
I20260926 14:53:51.482797 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:51.483106 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:51.484248 17290 hybrid_clock.cc:648] HybridClock initialized: now 1790434431484236 us; error 28 us; skew 500 ppm
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:51.486836 17290 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:53:51.487282 17290 webserver.cc:460] Webserver started at http://127.5.15.190:36239/ using document root <none> and password file <none>
I20260926 14:53:51.487550 17290 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:51.487624 17290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:51.545590 17290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:53:51.546391 17306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.546592 17290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:53:51.546690 17290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "faacac3ef86d4aca95312d01f061ce83"
format_stamp: "Formatted at 2026-09-26 14:53:49 on dist-test-slave-s4p1"
server_key: "1a80ab83281044070af34c3c8e3e9c74"
server_key_iv: "f1eb03c5b42d842cb54d61a8c284fbdb"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.547169 17290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:51.585585 17290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:51.587357 17290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:51.587467 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:51.591635 17290 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:36117
I20260926 14:53:51.591674 17358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:36117 every 8 connection(s)
I20260926 14:53:51.592027 17290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:51.593253 17359 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:53:51.593914 17359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Bootstrap starting.
I20260926 14:53:51.600008 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17290
I20260926 14:53:51.600302 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:42013
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:36117
--webserver_port=35511
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:51.601910 17359 log.cc:824] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:51.603662 17359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:53:51.603935 17359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Bootstrap complete.
I20260926 14:53:51.606247 17359 raft_consensus.cc:348] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:51.606494 17359 raft_consensus.cc:729] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faacac3ef86d4aca95312d01f061ce83, State: Initialized, Role: FOLLOWER
I20260926 14:53:51.606691 17359 consensus_queue.cc:260] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:51.606803 17359 raft_consensus.cc:388] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:51.606860 17359 raft_consensus.cc:482] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:51.606910 17359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:53:51.607859 17359 raft_consensus.cc:504] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:51.607991 17359 leader_election.cc:302] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faacac3ef86d4aca95312d01f061ce83; no voters:
I20260926 14:53:51.608178 17359 leader_election.cc:288] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:53:51.608214 17363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:53:51.608381 17363 raft_consensus.cc:686] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [term 2 LEADER]: Becoming Leader. State: Replica: faacac3ef86d4aca95312d01f061ce83, State: Running, Role: LEADER
I20260926 14:53:51.608479 17359 sys_catalog.cc:564] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:51.608532 17363 consensus_queue.cc:237] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } }
I20260926 14:53:51.608901 17363 sys_catalog.cc:455] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faacac3ef86d4aca95312d01f061ce83. Latest consensus state: current_term: 2 leader_uuid: "faacac3ef86d4aca95312d01f061ce83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } } }
I20260926 14:53:51.608906 17364 sys_catalog.cc:455] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "faacac3ef86d4aca95312d01f061ce83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faacac3ef86d4aca95312d01f061ce83" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36117 } } }
I20260926 14:53:51.609102 17364 sys_catalog.cc:458] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:51.609259 17363 sys_catalog.cc:458] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:51.609313 17372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:51.609834 17372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:51.609984 17372 catalog_manager.cc:1259] Loaded cluster ID: dc4ba89692494de59a6e85625cdea078
I20260926 14:53:51.610069 17372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:51.610265 17359 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:53:51.613754 17372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:51.614290 17372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Loaded TSK: 0
I20260926 14:53:51.614564 17372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:53:51.710633 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:51.711004 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:51.711090 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:51.712797 17361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:51.712898 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:51.712973 17361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:53:51.715167 17361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:42013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=35511
--tserver_master_addrs=127.5.15.190:36117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:51.715631 17361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:51.715951 17361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:51.722302 17399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.722421 17397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.722674 17361 server_base.cc:1034] running on GCE node
W20260926 14:53:51.722860 17396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.723116 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:51.723385 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:51.724570 17361 hybrid_clock.cc:648] HybridClock initialized: now 1790434431724545 us; error 34 us; skew 500 ppm
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:51.727898 17361 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:53:51.728418 17361 webserver.cc:460] Webserver started at http://127.5.15.129:35511/ using document root <none> and password file <none>
I20260926 14:53:51.728680 17361 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:51.728775 17361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:51.749485 17361 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.001s
I20260926 14:53:51.750571 17407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.750819 17361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:53:51.750901 17361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "d073945ab81949049b1a5167b1ffad47"
format_stamp: "Formatted at 2026-09-26 14:53:50 on dist-test-slave-s4p1"
server_key: "a4a5fa1fee84f10a9dcf8344dc3fdef7"
server_key_iv: "42888ba34c4b1112f1702d256b6dbb60"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.751377 17361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:51.784286 17361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:51.786463 17361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:51.786688 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:51.786976 17361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:51.787379 17361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:51.787425 17361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.787457 17361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:51.787511 17361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.794679 17361 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:42013
I20260926 14:53:51.795197 17520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:42013 every 8 connection(s)
I20260926 14:53:51.795982 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17361
I20260926 14:53:51.796267 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:39215
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:36117
--webserver_port=34463
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:51.795264 17361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:51.814941 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.802786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36117 (local address 127.5.15.129:47107)
0926 14:53:51.803503 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.803536 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:51.803986 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:51.804744 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:51.804748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:51.804969 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:51.805157 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.805163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.806789 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.806790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.807587 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.807589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.808520 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.808522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.808639 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.809271 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:51.809282 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:51.810920 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:51.813167 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.813173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.813174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.813446 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.813726 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.813727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.813727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.813850 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.814329 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:51.814334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:51.814475 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 14:53:51.814573 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:53:51.814719 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":77,"threads_started":1}
I20260926 14:53:51.815459 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.803165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47107 (local address 127.5.15.190:36117)
0926 14:53:51.804200 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:51.804209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:51.804247 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:53:51.804388 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:51.804394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:51.804445 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:51.804579 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:51.805274 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.806669 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.807718 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.808319 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.808421 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.811042 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:51.811046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:51.811047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:51.811077 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:51.813066 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.813535 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.813538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.813538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.813655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.813923 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.813924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.813925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.814222 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:51.814300 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:51.815175 (+ 875us) server_negotiation.cc:299] Negotiation successful
0926 14:53:51.815288 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":66,"threads_started":1}
I20260926 14:53:51.816180 17521 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36117
I20260926 14:53:51.816275 17521 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.816452 17521 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
W20260926 14:53:51.817241 17521 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:36117 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:53:51.817325 17521 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.817399 17521 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:51.817689 17521 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.817811 17521 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
W20260926 14:53:51.818122 17521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:53:51.906984 17523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:51.907310 17523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:51.907399 17523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:51.909075 17523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:51.909196 17523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:51.909274 17523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:53:51.911314 17523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:39215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=34463
--tserver_master_addrs=127.5.15.190:36117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:51.911744 17523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:51.912038 17523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:51.918385 17531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.918689 17523 server_base.cc:1034] running on GCE node
W20260926 14:53:51.918469 17530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:51.918412 17533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:51.919092 17523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:51.919348 17523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:51.920518 17523 hybrid_clock.cc:648] HybridClock initialized: now 1790434431920500 us; error 43 us; skew 500 ppm
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:51.923780 17523 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:53:51.924368 17523 webserver.cc:460] Webserver started at http://127.5.15.130:34463/ using document root <none> and password file <none>
I20260926 14:53:51.924603 17523 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:51.924687 17523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:51.938941 17523 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:53:51.939967 17541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.940155 17523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:53:51.940248 17523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "22ece3465b6744ebafd451ad7540df73"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "189b71d416f6a23499cd28d10a3b7737"
server_key_iv: "4fe7505a54e979dca7df523c94411388"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:51.940632 17523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:51.967563 17523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:51.969743 17523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:51.969913 17523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:51.970175 17523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:51.970485 17523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:51.970537 17523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.970580 17523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:51.970602 17523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:51.977339 17523 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:39215
I20260926 14:53:51.977428 17654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:39215 every 8 connection(s)
I20260926 14:53:51.977761 17523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:53:51 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:51.984732 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17523
I20260926 14:53:51.985025 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.15.148:42529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:46467
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:36117
--webserver_port=45459
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:51.987265 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.978537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:38431 (local address 127.5.15.190:36117)
0926 14:53:51.978580 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:51.978582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:51.978831 (+ 249us) server_negotiation.cc:407] Connection header received
0926 14:53:51.979362 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:51.979363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:51.979381 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:51.979460 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:51.980025 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.980580 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.981628 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.981839 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.981921 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.984269 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:51.984270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:51.984270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:51.984277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:51.985753 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.986159 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.986160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.986160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.986217 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:51.986477 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:51.986478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:51.986478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:51.986591 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:51.986658 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:51.987100 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 14:53:51.987182 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:53:51.987450 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:51.978534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36117 (local address 127.5.15.130:38431)
0926 14:53:51.978687 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:51.978722 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:51.979271 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:51.979517 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:51.979520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:51.979698 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:51.979937 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.979944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.980681 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.980682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.981525 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:51.981528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:51.981948 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:51.981950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:51.982076 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:51.982704 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:51.982714 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:51.984142 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:51.985854 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.985860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.985862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.986082 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.986297 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:51.986297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:51.986298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:51.986399 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:51.986661 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:51.986667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:51.986956 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 14:53:51.987078 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 14:53:51.987213 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260926 14:53:51.987890 17655 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36117
I20260926 14:53:51.987991 17655 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.988163 17655 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
W20260926 14:53:51.988765 17655 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:36117 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:53:51.988842 17655 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.988926 17655 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:51.989205 17655 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:51.989276 17655 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
W20260926 14:53:51.989552 17655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:53:52.093808 17658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:52.094079 17658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:52.094113 17658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:52.095744 17658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:52.095858 17658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:52.095949 17658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:53:52.098122 17658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.15.148:42529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:46467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=45459
--tserver_master_addrs=127.5.15.190:36117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:52.098663 17658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:52.098991 17658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:52.105337 17665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:52.105939 17664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:52.106176 17658 server_base.cc:1034] running on GCE node
W20260926 14:53:52.106518 17667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:52.106793 17658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:52.107046 17658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:52.108199 17658 hybrid_clock.cc:648] HybridClock initialized: now 1790434432108180 us; error 32 us; skew 500 ppm
Sep 26 14:53:52 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434432, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:52.111543 17658 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:53:52.112016 17658 webserver.cc:460] Webserver started at http://127.5.15.131:45459/ using document root <none> and password file <none>
I20260926 14:53:52.112268 17658 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:52.112331 17658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:53:52 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434432, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:52.130949 17658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260926 14:53:52.132100 17675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:52.132383 17658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:53:52.132527 17658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "d11088afa6b147a098190de516cef61e"
format_stamp: "Formatted at 2026-09-26 14:53:51 on dist-test-slave-s4p1"
server_key: "5c1ee730b8f02d4d09d94e61f04bf63a"
server_key_iv: "94dea809ac48cea9429851129ac30548"
server_key_version: "kuduclusterkey@0"
I20260926 14:53:52.133093 17658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:52.165028 17658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:52.167342 17658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:52.167490 17658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:52.167790 17658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:52.168160 17658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:52.168218 17658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:52.168288 17658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:52.168318 17658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:52.175374 17658 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46467
I20260926 14:53:52.175757 17658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:53:52.176697 17788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46467 every 8 connection(s)
Sep 26 14:53:52 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434432, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:52.184991 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17658
I20260926 14:53:52.188275 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:52.176957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:56125 (local address 127.5.15.190:36117)
0926 14:53:52.177049 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:52.177052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:52.177415 (+ 363us) server_negotiation.cc:407] Connection header received
0926 14:53:52.178127 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:52.178129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:52.178144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:52.178179 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:52.179028 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:52.179812 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:52.181272 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:52.181543 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:52.181614 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:52.184282 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:52.184284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:52.184284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:52.184292 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:52.186649 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:52.187098 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:52.187100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:52.187100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:52.187161 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:52.187464 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:52.187466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:52.187466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:52.187630 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:52.187708 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:52.188070 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 14:53:52.188182 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:53:52.188457 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:52.176547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36117 (local address 127.5.15.131:56125)
0926 14:53:52.177222 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:52.177333 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:52.177991 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:52.178378 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:52.178382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:52.178605 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:52.178912 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:52.178918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:52.179932 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:52.179933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:52.181147 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:52.181149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:52.181728 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:52.181730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:52.181885 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:52.182455 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:52.182464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:52.184169 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:52.186775 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:52.186782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:52.186784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:52.187007 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:52.187248 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:52.187249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:52.187250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:52.187374 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:52.187713 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:52.187719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:52.187883 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 14:53:52.188004 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 14:53:52.188174 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":76,"threads_started":1}
I20260926 14:53:52.189025 17789 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36117
I20260926 14:53:52.189117 17789 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:52.189287 17789 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
W20260926 14:53:52.189903 17789 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:36117 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:53:52.189972 17789 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:52.190029 17789 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:52.190408 17789 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:52.190495 17789 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
W20260926 14:53:52.190798 17789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:53:52.194868 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:52.185403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.5.15.190:36117)
0926 14:53:52.186449 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:52.186451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:52.187423 (+ 972us) server_negotiation.cc:407] Connection header received
0926 14:53:52.187441 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:52.187441 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:52.187450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:52.187523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:52.188564 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:52.189315 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:52.190425 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:52.190737 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:52.190833 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:52.192175 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:52.192176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:52.192176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:52.192182 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:52.193714 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:52.194069 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:52.194070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:52.194071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:52.194131 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:52.194348 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:52.194348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:52.194349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:52.194453 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:52.194629 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:52.194664 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 14:53:52.194739 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1023,"thread_start_us":70,"threads_started":1}
Sep 26 14:53:52 dist-test-slave-s4p1 krb5kdc[16327](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434432, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:52.818288 17521 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:52.818451 17521 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:52.818876 17323 ts_manager.cc:194] Registered new tserver with Master: d073945ab81949049b1a5167b1ffad47 (127.5.15.129:42013)
I20260926 14:53:52.819579 17323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:47107
I20260926 14:53:52.989714 17655 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:52.989868 17655 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:52.990154 17323 ts_manager.cc:194] Registered new tserver with Master: 22ece3465b6744ebafd451ad7540df73 (127.5.15.130:39215)
I20260926 14:53:52.990657 17323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:38431
Sep 26 14:53:53 dist-test-slave-s4p1 krb5kdc[16327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434432, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for HTTP/127.5.15.148@KRBTEST.COM
I20260926 14:53:53.190975 17789 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:53.191120 17789 heartbeater.cc:511] Master 127.5.15.190:36117 requested a full tablet report, sending...
I20260926 14:53:53.191397 17323 ts_manager.cc:194] Registered new tserver with Master: d11088afa6b147a098190de516cef61e (127.5.15.131:46467)
I20260926 14:53:53.191910 17323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:56125
I20260926 14:53:53.200325 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:53:53.211195 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.203104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.5.15.190:36117)
0926 14:53:53.203242 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.203244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.203396 (+ 152us) server_negotiation.cc:407] Connection header received
0926 14:53:53.203451 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.203452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.203464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.203535 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:53.204316 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.205103 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.206145 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.206432 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.206535 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.207958 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:53.207959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:53.207959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:53.207966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:53.209935 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:53.210335 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:53.210336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:53.210337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:53.210401 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:53.210624 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:53.210625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:53.210625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:53.210780 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:53.210882 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.210990 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.211094 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 14:53:53.213371 17323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:53:53.215530 17323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:53:53.223785 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.218393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42013 (local address 127.0.0.1:58846)
0926 14:53:53.219329 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:53.219337 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:53.219372 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:53.219748 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:53.219750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:53.219752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:53.219806 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.219808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.221085 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.221085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.222308 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.222310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.223507 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.223508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.223587 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.223599 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 14:53:53.223662 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:53:53.223715 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":88,"threads_started":1}
I20260926 14:53:53.224265 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.218642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.5.15.129:42013)
0926 14:53:53.218801 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.218806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.219427 (+ 621us) server_negotiation.cc:407] Connection header received
0926 14:53:53.219465 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.219468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.219484 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.219550 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:53.219872 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.221007 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.222404 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.223334 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.223402 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.223473 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.224156 (+ 683us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.224197 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
I20260926 14:53:53.225014 17455 tablet_service.cc:1467] Processing CreateTablet for tablet e40910e114cd4f13b43a4155ab296a0f (DEFAULT_TABLE table=test-table [id=bee41bdf923042c787dba9bf95fb57c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:53.225353 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40910e114cd4f13b43a4155ab296a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:53.226828 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.218387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:39215 (local address 127.0.0.1:49488)
0926 14:53:53.219146 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:53.219184 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:53.219239 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:53.223022 (+ 3783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:53.223023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:53.223025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:53.223068 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.223070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.224389 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.224389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.225431 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.225433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.226607 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.226608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.226669 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.226672 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:53:53.226699 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 14:53:53.226733 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":85,"threads_started":1}
I20260926 14:53:53.227125 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.218905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.5.15.130:39215)
0926 14:53:53.222857 (+ 3952us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.222862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.222871 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:53:53.222902 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.222904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.222918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.223130 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:53.223363 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.224299 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.225526 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.226516 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.226917 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.227001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.227025 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.227056 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3921,"thread_start_us":64,"threads_started":1}
I20260926 14:53:53.229938 17575 tablet_service.cc:1467] Processing CreateTablet for tablet e40910e114cd4f13b43a4155ab296a0f (DEFAULT_TABLE table=test-table [id=bee41bdf923042c787dba9bf95fb57c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:53.230294 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40910e114cd4f13b43a4155ab296a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:53.230926 17822 tablet_bootstrap.cc:492] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47: Bootstrap starting.
I20260926 14:53:53.231194 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.218614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46467 (local address 127.0.0.1:41680)
0926 14:53:53.219640 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:53.219649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:53.219685 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:53.227450 (+ 7765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:53.227451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:53.227453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:53.227488 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.227490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.228729 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.228730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.229836 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.229838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.231010 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.231011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.231068 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.231072 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:53:53.231099 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 14:53:53.231131 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":73,"threads_started":1}
I20260926 14:53:53.231627 17822 tablet_bootstrap.cc:654] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:53.231711 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.219055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.5.15.131:46467)
0926 14:53:53.227289 (+ 8234us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.227294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.227303 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:53:53.227334 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.227336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.227351 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.227552 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:53.227618 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.228637 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.229992 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.230897 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.231484 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.231578 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.231605 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.231641 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8205,"thread_start_us":47,"threads_started":1}
I20260926 14:53:53.232009 17822 log.cc:824] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:53.232474 17713 tablet_service.cc:1467] Processing CreateTablet for tablet e40910e114cd4f13b43a4155ab296a0f (DEFAULT_TABLE table=test-table [id=bee41bdf923042c787dba9bf95fb57c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:53:53.232764 17713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e40910e114cd4f13b43a4155ab296a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:53.233695 17824 tablet_bootstrap.cc:492] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Bootstrap starting.
I20260926 14:53:53.234301 17824 tablet_bootstrap.cc:654] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:53.234635 17824 log.cc:824] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:53.235870 17826 tablet_bootstrap.cc:492] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Bootstrap starting.
I20260926 14:53:53.236475 17826 tablet_bootstrap.cc:654] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:53.236747 17826 log.cc:824] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:53.237533 17826 tablet_bootstrap.cc:492] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: No bootstrap required, opened a new log
I20260926 14:53:53.237602 17826 ts_tablet_manager.cc:1397] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:53:53.239320 17822 tablet_bootstrap.cc:492] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47: No bootstrap required, opened a new log
I20260926 14:53:53.239390 17822 ts_tablet_manager.cc:1397] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 14:53:53.239821 17826 raft_consensus.cc:348] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.239974 17826 raft_consensus.cc:374] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:53.240010 17826 raft_consensus.cc:729] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11088afa6b147a098190de516cef61e, State: Initialized, Role: FOLLOWER
I20260926 14:53:53.240120 17826 consensus_queue.cc:260] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.240361 17826 ts_tablet_manager.cc:1428] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:53:53.240474 17789 heartbeater.cc:503] Master 127.5.15.190:36117 was elected leader, sending a full tablet report...
I20260926 14:53:53.241647 17822 raft_consensus.cc:348] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.241775 17822 raft_consensus.cc:374] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:53.241802 17822 raft_consensus.cc:729] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d073945ab81949049b1a5167b1ffad47, State: Initialized, Role: FOLLOWER
I20260926 14:53:53.241900 17822 consensus_queue.cc:260] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.242141 17822 ts_tablet_manager.cc:1428] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:53:53.242209 17521 heartbeater.cc:503] Master 127.5.15.190:36117 was elected leader, sending a full tablet report...
I20260926 14:53:53.243109 17824 tablet_bootstrap.cc:492] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: No bootstrap required, opened a new log
I20260926 14:53:53.243175 17824 ts_tablet_manager.cc:1397] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260926 14:53:53.245195 17829 raft_consensus.cc:482] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:53.245270 17829 raft_consensus.cc:504] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.245383 17824 raft_consensus.cc:348] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.245510 17824 raft_consensus.cc:374] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:53.245536 17824 raft_consensus.cc:729] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ece3465b6744ebafd451ad7540df73, State: Initialized, Role: FOLLOWER
I20260926 14:53:53.245627 17824 consensus_queue.cc:260] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.245888 17824 ts_tablet_manager.cc:1428] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:53:53.246201 17655 heartbeater.cc:503] Master 127.5.15.190:36117 was elected leader, sending a full tablet report...
I20260926 14:53:53.246241 17829 leader_election.cc:288] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22ece3465b6744ebafd451ad7540df73 (127.5.15.130:39215), d11088afa6b147a098190de516cef61e (127.5.15.131:46467)
I20260926 14:53:53.249912 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.245885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:45553 (local address 127.5.15.130:39215)
0926 14:53:53.245924 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.245926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.246417 (+ 491us) server_negotiation.cc:407] Connection header received
0926 14:53:53.246535 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.246536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.246546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.246574 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:53.246818 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.247444 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.248796 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.249364 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.249414 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.249421 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.249810 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.249852 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:53:53.250142 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.245672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:39215 (local address 127.5.15.129:45553)
0926 14:53:53.246349 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:53.246360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:53.246444 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:53.246634 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:53.246635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:53.246637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:53.246698 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.246700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.247556 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.247557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.248692 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.248695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.249652 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.249653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.249720 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.249724 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:53:53.250010 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 14:53:53.250054 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":134,"threads_started":1}
I20260926 14:53:53.250406 17609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40910e114cd4f13b43a4155ab296a0f" candidate_uuid: "d073945ab81949049b1a5167b1ffad47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ece3465b6744ebafd451ad7540df73" is_pre_election: true
I20260926 14:53:53.250525 17609 raft_consensus.cc:2454] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d073945ab81949049b1a5167b1ffad47 in term 0.
I20260926 14:53:53.250818 17410 leader_election.cc:302] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22ece3465b6744ebafd451ad7540df73, d073945ab81949049b1a5167b1ffad47; no voters:
I20260926 14:53:53.250845 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.246191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47661 (local address 127.5.15.131:46467)
0926 14:53:53.246223 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.246225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.247272 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 14:53:53.247354 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.247355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.247362 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.247387 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:53:53.247560 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.248212 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.249515 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.250058 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.250099 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.250105 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.250710 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.250759 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:53:53.251041 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.246036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46467 (local address 127.5.15.129:47661)
0926 14:53:53.247219 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:53.247228 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:53.247294 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:53.247434 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:53.247435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:53.247436 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:53:53.247496 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.247497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.248293 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.248294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.249420 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:53.249422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.250542 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.250543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:53.250612 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.250615 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:53:53.250910 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 14:53:53.250948 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":64,"threads_started":1}
I20260926 14:53:53.251214 17829 raft_consensus.cc:2781] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:53.251277 17829 raft_consensus.cc:482] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:53.251302 17829 raft_consensus.cc:3037] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:53.252020 17829 raft_consensus.cc:504] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.252120 17829 leader_election.cc:288] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [CANDIDATE]: Term 1 election: Requested vote from peers 22ece3465b6744ebafd451ad7540df73 (127.5.15.130:39215), d11088afa6b147a098190de516cef61e (127.5.15.131:46467)
I20260926 14:53:53.252282 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40910e114cd4f13b43a4155ab296a0f" candidate_uuid: "d073945ab81949049b1a5167b1ffad47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11088afa6b147a098190de516cef61e" is_pre_election: true
I20260926 14:53:53.252362 17609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40910e114cd4f13b43a4155ab296a0f" candidate_uuid: "d073945ab81949049b1a5167b1ffad47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ece3465b6744ebafd451ad7540df73"
I20260926 14:53:53.252381 17739 raft_consensus.cc:2454] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d073945ab81949049b1a5167b1ffad47 in term 0.
I20260926 14:53:53.252437 17609 raft_consensus.cc:3037] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:53.252494 17737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e40910e114cd4f13b43a4155ab296a0f" candidate_uuid: "d073945ab81949049b1a5167b1ffad47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11088afa6b147a098190de516cef61e"
I20260926 14:53:53.252547 17737 raft_consensus.cc:3037] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:53.253186 17609 raft_consensus.cc:2454] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d073945ab81949049b1a5167b1ffad47 in term 1.
I20260926 14:53:53.253186 17737 raft_consensus.cc:2454] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d073945ab81949049b1a5167b1ffad47 in term 1.
I20260926 14:53:53.253391 17410 leader_election.cc:302] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d073945ab81949049b1a5167b1ffad47, d11088afa6b147a098190de516cef61e; no voters:
I20260926 14:53:53.253500 17829 raft_consensus.cc:2781] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:53.253648 17829 raft_consensus.cc:686] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [term 1 LEADER]: Becoming Leader. State: Replica: d073945ab81949049b1a5167b1ffad47, State: Running, Role: LEADER
I20260926 14:53:53.253968 17829 consensus_queue.cc:237] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } }
I20260926 14:53:53.254786 17322 catalog_manager.cc:5495] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 reported cstate change: term changed from 0 to 1, leader changed from <none> to d073945ab81949049b1a5167b1ffad47 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "d073945ab81949049b1a5167b1ffad47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d073945ab81949049b1a5167b1ffad47" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42013 } health_report { overall_health: HEALTHY } } }
I20260926 14:53:53.263242 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.260353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.5.15.129:42013)
0926 14:53:53.260426 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:53.260427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:53.260436 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:53:53.260472 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:53.260473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:53.260490 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:53.260564 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:53:53.260702 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.261341 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.262397 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:53.262689 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:53.262784 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:53.262940 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:53:53.263042 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:53:53.263124 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:53.263165 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:53:53.263183 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:53:53.264902 17737 raft_consensus.cc:1261] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 1 FOLLOWER]: Refusing update from remote peer d073945ab81949049b1a5167b1ffad47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:53.264921 17609 raft_consensus.cc:1261] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 1 FOLLOWER]: Refusing update from remote peer d073945ab81949049b1a5167b1ffad47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:53.265240 17833 consensus_queue.cc:1035] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ece3465b6744ebafd451ad7540df73" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:53.265379 17829 consensus_queue.cc:1035] T e40910e114cd4f13b43a4155ab296a0f P d073945ab81949049b1a5167b1ffad47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11088afa6b147a098190de516cef61e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:53.267797 17836 mvcc.cc:205] Tried to move back new op lower bound from 7333619438650642432 to 7333619438609338368. Current Snapshot: MvccSnapshot[applied={T|T < 7333619438650642432}]
I20260926 14:53:53.268544 17838 mvcc.cc:205] Tried to move back new op lower bound from 7333619438650642432 to 7333619438609338368. Current Snapshot: MvccSnapshot[applied={T|T < 7333619438650642432}]
I20260926 14:53:53.272876 17322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:53.272965 17322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:53:53.273664 17322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faacac3ef86d4aca95312d01f061ce83: Sending DeleteTablet for 3 replicas of tablet e40910e114cd4f13b43a4155ab296a0f
I20260926 14:53:53.274039 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet e40910e114cd4f13b43a4155ab296a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:53 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:49488
I20260926 14:53:53.274190 17713 tablet_service.cc:1514] Processing DeleteTablet for tablet e40910e114cd4f13b43a4155ab296a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:53 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:41680
I20260926 14:53:53.274379 17455 tablet_service.cc:1514] Processing DeleteTablet for tablet e40910e114cd4f13b43a4155ab296a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:53:53 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:58846
I20260926 14:53:53.274407 17846 tablet_replica.cc:330] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: stopping tablet replica
I20260926 14:53:53.274502 17846 raft_consensus.cc:2229] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:53.274595 17846 raft_consensus.cc:2258] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:53.274786 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17361
I20260926 14:53:53.275657 17846 ts_tablet_manager.cc:1905] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:53.277297 17846 ts_tablet_manager.cc:1918] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:53.277360 17846 log.cc:1196] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/e40910e114cd4f13b43a4155ab296a0f
I20260926 14:53:53.277657 17846 ts_tablet_manager.cc:1939] T e40910e114cd4f13b43a4155ab296a0f P d11088afa6b147a098190de516cef61e: Deleting consensus metadata
I20260926 14:53:53.278095 17309 catalog_manager.cc:4841] TS d11088afa6b147a098190de516cef61e (127.5.15.131:46467): tablet e40910e114cd4f13b43a4155ab296a0f (table test-table [id=bee41bdf923042c787dba9bf95fb57c7]) successfully deleted
I20260926 14:53:53.278884 17845 tablet_replica.cc:330] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: stopping tablet replica
I20260926 14:53:53.279037 17845 raft_consensus.cc:2229] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:53:53.279182 17845 raft_consensus.cc:2258] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:53:53.279651 17845 ts_tablet_manager.cc:1905] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:53:53.281229 17845 ts_tablet_manager.cc:1918] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:53:53.281344 17845 log.cc:1196] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/e40910e114cd4f13b43a4155ab296a0f
I20260926 14:53:53.281668 17845 ts_tablet_manager.cc:1939] T e40910e114cd4f13b43a4155ab296a0f P 22ece3465b6744ebafd451ad7540df73: Deleting consensus metadata
I20260926 14:53:53.282097 17309 catalog_manager.cc:4841] TS 22ece3465b6744ebafd451ad7540df73 (127.5.15.130:39215): tablet e40910e114cd4f13b43a4155ab296a0f (table test-table [id=bee41bdf923042c787dba9bf95fb57c7]) successfully deleted
W20260926 14:53:53.282403 17307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:42013 (error 108)
I20260926 14:53:53.282853 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:53.282724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42013 (local address 127.0.0.1:58866)
0926 14:53:53.282779 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:53.282811 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:42013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:53:53.282953 17307 catalog_manager.cc:4584] TS d073945ab81949049b1a5167b1ffad47 (127.5.15.129:42013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e40910e114cd4f13b43a4155ab296a0f: Network error: Client connection negotiation failed: client connection to 127.5.15.129:42013: connect: Connection refused (error 111)
I20260926 14:53:53.283253 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17523
I20260926 14:53:53.287213 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17658
I20260926 14:53:53.291568 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17290
I20260926 14:53:53.296350 5182 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:53:53.641139 5182 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:53:53.641237 5182 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:53:53.985611 5182 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:53:53.985 UTC [16357] LOG: received smart shutdown request
2026-09-26 14:53:53.988 UTC [16357] LOG: background worker "logical replication launcher" (PID 16366) exited with exit code 1
2026-09-26 14:53:53.989 UTC [16361] LOG: shutting down
2026-09-26 14:53:54.143 UTC [16357] LOG: database system is shut down
2026-09-26T14:53:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44840 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:54 dist-test-slave-s4p1 krb5kdc[17857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:54 dist-test-slave-s4p1 krb5kdc[17857](info): set up 2 sockets
Sep 26 14:53:54 dist-test-slave-s4p1 krb5kdc[17857](info): commencing operation
krb5kdc: starting...
W20260926 14:53:55.266611 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:53:55.377751 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41583
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:41583,127.5.15.189:39715,127.5.15.188:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:55.455796 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.456041 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.456079 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.457198 17870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:55.457268 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.457290 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.457311 17870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:55.457341 17870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:55.458846 17870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:41583,127.5.15.189:39715,127.5.15.188:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:55.459115 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.459344 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:55.464059 17878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.464088 17876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.464073 17875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.464361 17870 server_base.cc:1034] running on GCE node
I20260926 14:53:55.464648 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.464939 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.466114 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790434435466099 us; error 27 us; skew 500 ppm
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:55.472486 17870 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:53:55.472918 17870 webserver.cc:460] Webserver started at http://127.5.15.190:37429/ using document root <none> and password file <none>
I20260926 14:53:55.473160 17870 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.473228 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.473376 17870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.474460 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "d768e828db1a49e88b404bdffecb5842"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "874cd1b10a2f764c648756dbf70412f6"
server_key_iv: "720b2ba4f4869437b89c0b027c928daf"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.474891 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "d768e828db1a49e88b404bdffecb5842"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "874cd1b10a2f764c648756dbf70412f6"
server_key_iv: "720b2ba4f4869437b89c0b027c928daf"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.476202 17870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:53:55.476925 17885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.477099 17870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:53:55.477196 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "d768e828db1a49e88b404bdffecb5842"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "874cd1b10a2f764c648756dbf70412f6"
server_key_iv: "720b2ba4f4869437b89c0b027c928daf"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.477283 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.503485 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.505383 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.505542 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.509160 17870 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:41583
I20260926 14:53:55.509202 17937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:41583 every 8 connection(s)
I20260926 14:53:55.509577 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:55.510448 17938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.512166 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17870
I20260926 14:53:55.512245 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:53:55.512032 17938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:53:55.512421 5182 external_mini_cluster.cc:1428] Setting key ad66fb9b20055c664ead7cf1dd2e38dc
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.15.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.189@KRBTEST.COM" created.
I20260926 14:53:55.521771 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.512777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41583 (local address 127.0.0.1:41154)
0926 14:53:55.512969 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.513012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.513705 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.513974 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.513977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.514219 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.514365 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.514369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.515430 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.515431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.516146 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.516148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.516634 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.516635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.516696 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.517216 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.517224 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.518521 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.520340 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.520346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.520348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.520535 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.520784 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.520785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.520785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.520871 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.521207 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.521212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.521270 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.521375 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.521599 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":100,"thread_start_us":83,"threads_started":1}
I20260926 14:53:55.521773 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.512835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.5.15.190:41583)
0926 14:53:55.512974 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.512983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.513115 (+ 132us) server_negotiation.cc:407] Connection header received
0926 14:53:55.513843 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.513845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.513861 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.513932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.514478 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.515324 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.516247 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.516544 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.516652 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.518609 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.518615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.518616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.518622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.520235 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.520628 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.520630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.520631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.520714 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.520937 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.520938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.520939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.521131 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.521226 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.521411 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.521539 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260926 14:53:55.522670 17938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:53:55.523026 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.522910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51448)
0926 14:53:55.522961 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.523008 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:53:55.523173 17888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
I20260926 14:53:55.523634 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.523543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51464)
0926 14:53:55.523590 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.523613 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:53:55.523778 17938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:39715: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab.
Entry for principal kudu/127.5.15.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab.
I20260926 14:53:55.536748 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:39715
--webserver_interface=127.5.15.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab
--principal=kudu/127.5.15.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:41583,127.5.15.189:39715,127.5.15.188:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:55.573117 17938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } attempt: 1
I20260926 14:53:55.573630 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.573486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51466)
0926 14:53:55.573563 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.573601 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 14:53:55.573982 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.573885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51474)
0926 14:53:55.573930 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.573954 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 14:53:55.574065 17938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:39715: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
W20260926 14:53:55.614286 17945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.614467 17945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.614488 17945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.615610 17945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:55.615648 17945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.615662 17945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.615674 17945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:55.615726 17945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:55.617075 17945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:41583,127.5.15.189:39715,127.5.15.188:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab
--principal=kudu/127.5.15.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:39715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.15.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:55.617239 17945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.617453 17945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:55.622001 17951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.622001 17950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.622014 17953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.622247 17945 server_base.cc:1034] running on GCE node
I20260926 14:53:55.622457 17945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.622790 17945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.623958 17945 hybrid_clock.cc:648] HybridClock initialized: now 1790434435623950 us; error 39 us; skew 500 ppm
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:55.626276 17945 init.cc:377] Logged in from keytab as kudu/127.5.15.189@KRBTEST.COM (short username kudu)
I20260926 14:53:55.626734 17945 webserver.cc:460] Webserver started at http://127.5.15.189:39933/ using document root <none> and password file <none>
I20260926 14:53:55.626964 17945 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.627029 17945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.627157 17945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.628079 17945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data/instance:
uuid: "ad63ac7210a64748a112cd088640ed36"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "a003d92b523f1b9e9f96028552b374ae"
server_key_iv: "1f22d83d9ab16e90a4e70f43737ccbed"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.628417 17945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal/instance:
uuid: "ad63ac7210a64748a112cd088640ed36"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "a003d92b523f1b9e9f96028552b374ae"
server_key_iv: "1f22d83d9ab16e90a4e70f43737ccbed"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.629642 17945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:53:55.630286 17960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.630452 17945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:53:55.630568 17945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal
uuid: "ad63ac7210a64748a112cd088640ed36"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "a003d92b523f1b9e9f96028552b374ae"
server_key_iv: "1f22d83d9ab16e90a4e70f43737ccbed"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.630647 17945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.642222 17938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } attempt: 2
I20260926 14:53:55.642570 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.642456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51480)
0926 14:53:55.642510 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.642542 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:53:55.642913 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.642822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51490)
0926 14:53:55.642864 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.642888 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:53:55.642992 17938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:39715: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39715: connect: Connection refused (error 111)
I20260926 14:53:55.651609 17945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.653448 17945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.653614 17945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.657729 17945 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.189:39715
I20260926 14:53:55.657760 18012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.189:39715 every 8 connection(s)
I20260926 14:53:55.658119 17945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
I20260926 14:53:55.659075 18013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.660713 18013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:53:55.661084 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 17945
I20260926 14:53:55.661173 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-1/wal/instance
I20260926 14:53:55.661362 5182 external_mini_cluster.cc:1428] Setting key 8a29f301781531b4b5bc28af78995e84
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.15.188@KRBTEST.COM; defaulting to no policy
I20260926 14:53:55.670468 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.661541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.5.15.190:41583)
0926 14:53:55.661610 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.661612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.661840 (+ 228us) server_negotiation.cc:407] Connection header received
0926 14:53:55.662267 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.662269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.662281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.662361 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.662997 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.663774 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.664746 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.665122 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.665211 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.667611 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.667613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.667613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.667622 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.669239 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.669671 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.669673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.669673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.669736 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.669938 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.669939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.669939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.670038 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.670084 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.670308 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.670378 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:53:55.670691 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.661516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41583 (local address 127.0.0.1:41168)
0926 14:53:55.661708 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.661748 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.662182 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.662411 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.662414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.662654 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.662904 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.662909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.663867 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.663868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.664630 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.664631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.665211 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.665212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.665323 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.665866 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.665875 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.667494 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.669325 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.669331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.669332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.669598 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.669805 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.669806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.669806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.669896 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.670089 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.670093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.670201 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.670280 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.670393 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260926 14:53:55.671258 18013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.188@KRBTEST.COM" created.
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:53:55.679051 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.671533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51492)
0926 14:53:55.671586 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.671597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.671642 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.671909 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.671909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.672057 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.672129 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.672131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.672962 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.672963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.673761 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.673764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.674212 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.674213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.674288 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.674718 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.674723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.676231 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.677916 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.677916 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.677917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.678090 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.678316 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.678317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.678317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.678370 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.678669 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.678670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.678709 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.678828 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.678930 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:53:55.679131 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.671600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.5.15.189:39715)
0926 14:53:55.671761 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.671765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.671781 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:53:55.671799 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.671801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.671812 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.671872 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.672190 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.672861 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.673851 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.674133 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.674195 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.676327 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.676335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.676335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.676341 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.677809 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.678184 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.678187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.678187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.678247 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.678441 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.678442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.678443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.678592 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.678679 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.678859 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.679003 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 14:53:55.679685 18013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:53:55.680058 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.679958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51504)
0926 14:53:55.679997 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.680039 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 14:53:55.680197 17962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
I20260926 14:53:55.680590 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.680515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51508)
0926 14:53:55.680544 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.680568 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 14:53:55.680698 18013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:45483: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab.
Entry for principal kudu/127.5.15.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab.
I20260926 14:53:55.687886 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:45483
--webserver_interface=127.5.15.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab
--principal=kudu/127.5.15.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:41583,127.5.15.189:39715,127.5.15.188:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:53:55.729938 18013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } attempt: 1
I20260926 14:53:55.730417 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.730284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51520)
0926 14:53:55.730351 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.730389 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:53:55.730861 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.730766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51536)
0926 14:53:55.730806 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.730835 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 14:53:55.730971 18013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:45483: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
I20260926 14:53:55.734114 17938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } attempt: 3
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:53:55.741122 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.734420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.5.15.189:39715)
0926 14:53:55.734472 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.734474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.734484 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:53:55.734547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.734548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.734558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.734625 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.735014 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.735533 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.736231 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.736468 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.736545 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.738649 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.738650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.738650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.738656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.740133 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.740496 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.740497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.740497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.740546 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.740729 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.740730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.740730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.740822 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.740870 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.741019 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.741071 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:53:55.741204 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.734374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51500)
0926 14:53:55.734418 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.734428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.734482 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.734631 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.734632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.734849 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.734912 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.734914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.735615 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.735616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.736116 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.736117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.736543 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.736544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.736622 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.737095 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.737101 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.738568 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.740214 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.740215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.740216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.740422 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.740607 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.740607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.740608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.740671 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.740879 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.740880 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.740930 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.741003 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.741095 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 14:53:55.741648 17938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:53:55.741973 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.741868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51546)
0926 14:53:55.741916 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.741949 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:53:55.742336 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.742242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51552)
0926 14:53:55.742287 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.742314 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:53:55.742447 17938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:45483: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
W20260926 14:53:55.767184 18020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:55.767342 18020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:55.767360 18020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:55.768383 18020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:55.768414 18020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:55.768427 18020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:55.768440 18020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:55.768452 18020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:55.769788 18020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:41583,127.5.15.189:39715,127.5.15.188:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab
--principal=kudu/127.5.15.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.15.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:55.769955 18020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:55.770110 18020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:53:55.773604 17938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } attempt: 1
I20260926 14:53:55.774104 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.773931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51564)
0926 14:53:55.773999 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.774069 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:53:55.774435 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.774353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51578)
0926 14:53:55.774387 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.774408 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 14:53:55.774466 18026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.774477 18025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:55.774580 17938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:45483: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
W20260926 14:53:55.774478 18028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:55.774910 18020 server_base.cc:1034] running on GCE node
I20260926 14:53:55.775135 18020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:55.775427 18020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:55.776587 18020 hybrid_clock.cc:648] HybridClock initialized: now 1790434435776574 us; error 23 us; skew 500 ppm
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:55.779075 18020 init.cc:377] Logged in from keytab as kudu/127.5.15.188@KRBTEST.COM (short username kudu)
I20260926 14:53:55.779497 18020 webserver.cc:460] Webserver started at http://127.5.15.188:36789/ using document root <none> and password file <none>
I20260926 14:53:55.779738 18020 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:55.779805 18020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:55.779955 18020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:55.781051 18020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data/instance:
uuid: "a48aeb4b18d0449fbffad404cb4e4048"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "32982fce074956d45696dd1f1107dc23"
server_key_iv: "fc3d37f5fdf7753e36331955a2ec3da1"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.781412 18020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal/instance:
uuid: "a48aeb4b18d0449fbffad404cb4e4048"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "32982fce074956d45696dd1f1107dc23"
server_key_iv: "fc3d37f5fdf7753e36331955a2ec3da1"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.782858 18020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:53:55.783552 18035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:55.783725 18020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:53:55.783811 18020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal
uuid: "a48aeb4b18d0449fbffad404cb4e4048"
format_stamp: "Formatted at 2026-09-26 14:53:55 on dist-test-slave-s4p1"
server_key: "32982fce074956d45696dd1f1107dc23"
server_key_iv: "fc3d37f5fdf7753e36331955a2ec3da1"
server_key_version: "encryptionkey@0"
I20260926 14:53:55.783895 18020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:55.795898 18020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:55.798188 18020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:55.798354 18020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:55.799106 18013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } attempt: 2
I20260926 14:53:55.799526 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.799363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51590)
0926 14:53:55.799433 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.799483 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:53:55.799921 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.799813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51594)
0926 14:53:55.799861 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.799894 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:53:55.800064 18013 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:45483: Network error: Client connection negotiation failed: client connection to 127.5.15.188:45483: connect: Connection refused (error 111)
I20260926 14:53:55.802606 18020 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.188:45483
I20260926 14:53:55.802634 18087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.188:45483 every 8 connection(s)
I20260926 14:53:55.803022 18020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
I20260926 14:53:55.803869 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:55.805307 18088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:55.812588 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18020
I20260926 14:53:55.812685 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434201368912-5182-0/minicluster-data/master-2/wal/instance
I20260926 14:53:55.812875 5182 external_mini_cluster.cc:1428] Setting key 18b205e42d637cfe7cbcf7353b2df609
I20260926 14:53:55.814095 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.805903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:41583 (local address 127.0.0.1:41170)
0926 14:53:55.806075 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.806111 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.806554 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.806829 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.806832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.807086 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.807291 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.807297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.807931 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.807931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.808691 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.808692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.809010 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.809011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.809108 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.809564 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.809571 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.810904 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.812513 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.812521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.812523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.812836 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.813099 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.813099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.813100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.813214 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.813553 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.813556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.813721 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.813824 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.813936 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260926 14:53:55.814109 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.805941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.5.15.190:41583)
0926 14:53:55.806022 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.806025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.806189 (+ 164us) server_negotiation.cc:407] Connection header received
0926 14:53:55.806647 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.806647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.806656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.806708 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.807371 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.807854 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.808762 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.808938 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.808988 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.810983 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.810985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.810985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.810992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.812415 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.812956 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.812957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.812958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.813018 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.813307 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.813308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.813308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.813460 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.813538 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.813872 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.813961 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:53:55.814687 18088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:53:55.821622 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.813272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.5.15.188:45483)
0926 14:53:55.813432 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.813437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.813571 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:53:55.813595 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.813597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.813611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.813688 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.814463 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.815365 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.816085 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.816522 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.816592 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.818799 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.818805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.818806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.818812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.820400 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.820773 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.820776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.820776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.820823 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.821018 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.821019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.821020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.821191 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.821354 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.821400 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.821479 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":110,"threads_started":1}
I20260926 14:53:55.822012 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.815029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.5.15.189:39715)
0926 14:53:55.815079 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.815081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.815090 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:53:55.815112 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.815113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.815122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.815193 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.815452 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.815906 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.816820 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.817068 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.817142 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.819290 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.819291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.819292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.819298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.820836 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.821190 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.821191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.821191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.821261 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.821465 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.821466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.821466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.821614 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.821703 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.821861 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.821930 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:53:55.822041 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.814947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39715 (local address 127.0.0.1:51516)
0926 14:53:55.814993 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.815003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.815036 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.815195 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.815196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.815327 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.815373 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.815375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.815981 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.815982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.816728 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.816730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.817147 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.817147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.817194 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.817561 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.817566 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.819223 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.820922 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.820922 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.820923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.821119 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.821334 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.821334 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.821335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.821395 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.821709 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.821710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.821757 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.821835 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.821920 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:53:55.822453 18088 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:55.828583 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.822730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51612)
0926 14:53:55.822781 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.822792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.822825 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.822929 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.822930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.823087 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.823123 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.823124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.823698 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.823699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.824156 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.824157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.824522 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.824522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.824576 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.824906 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.824910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.825937 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.827585 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.827586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.827587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.827764 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.827977 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.827978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.827979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.828056 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.828287 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.828287 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.828322 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.828388 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.828476 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:53:55.828639 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.822782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.5.15.188:45483)
0926 14:53:55.822810 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.822812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.822819 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:53:55.822877 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.822877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.822885 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.822924 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.823189 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.823620 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.824245 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.824447 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.824513 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.826018 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.826019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.826019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.826026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.827487 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.827859 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.827860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.827860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.827907 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.828118 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.828119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.828119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.828222 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.828294 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.828415 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.828521 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":13}
I20260926 14:53:55.829133 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.822584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.5.15.190:41583)
0926 14:53:55.822623 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.822625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.822633 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:53:55.822664 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.822665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.822673 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.822718 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.823279 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.823714 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.824359 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.824556 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.824625 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.826627 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.826628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.826628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.826635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.827982 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.828335 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.828337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.828337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.828392 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.828578 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.828579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.828580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.828711 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.828902 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.828936 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.829004 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:53:55.830075 18088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048: Bootstrap starting.
I20260926 14:53:55.831131 18088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.831449 18088 log.cc:824] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.832185 18088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048: No bootstrap required, opened a new log
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:53:55.834405 18088 raft_consensus.cc:348] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.834533 18088 raft_consensus.cc:374] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.834563 18088 raft_consensus.cc:729] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48aeb4b18d0449fbffad404cb4e4048, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.834702 18088 consensus_queue.cc:260] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.835044 18088 sys_catalog.cc:564] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:55.835009 18094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.835096 18094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:53:55.836853 18105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:53:55.836956 18105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:53:55.837787 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.829781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.5.15.189:39715)
0926 14:53:55.829823 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.829824 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.829877 (+ 53us) server_negotiation.cc:407] Connection header received
0926 14:53:55.829946 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.829947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.829954 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.829992 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.830560 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.831051 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.831886 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.832063 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.832114 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.834699 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.834700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.834700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.834707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.836469 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.836879 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.836881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.836881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.836937 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.837148 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.837149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.837149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.837264 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.837411 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.837617 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.837694 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:53:55.837424 18020 master_runner.cc:186] Error getting master registration for 127.5.15.190:41583: OK, instance_id { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" instance_seqno: 1790434435508079 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:53:55.839257 18020 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:53:55.840691 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:53:55.849493 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.842088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51542 (local address 127.5.15.189:39715)
0926 14:53:55.842120 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.842122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.842223 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:53:55.842285 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.842286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.842296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.842353 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.843070 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.843749 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.844808 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.845101 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.845142 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.846809 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.846810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.846810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.846817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.848308 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.848667 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.848668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.848669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.848720 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.848903 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.848904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.848904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.849012 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.849110 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.849306 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.849402 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 14:53:55.849798 17938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } attempt: 2
I20260926 14:53:55.850123 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.842000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.5.15.188:45483)
0926 14:53:55.842037 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.842039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.842366 (+ 327us) server_negotiation.cc:407] Connection header received
0926 14:53:55.842457 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.842458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.842468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.842509 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.843233 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.843906 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.844922 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.845204 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.845256 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.846919 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.846920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.846920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.846927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.848566 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.848957 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.848958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.848959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.849012 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.849221 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.849222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.849222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.849389 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.849477 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.849955 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.850029 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:53:55.851826 17941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.841975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.5.15.190:41583)
0926 14:53:55.842013 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.842015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.842248 (+ 233us) server_negotiation.cc:407] Connection header received
0926 14:53:55.842347 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.842347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.842357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.842390 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.843161 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.843847 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.844922 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.845212 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.845260 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.847954 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.847955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.847956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.847963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.849614 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.851076 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.851077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.851078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.851141 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.851345 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.851346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.851346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.851450 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.851543 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.851648 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.851718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:53:55.856756 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.850239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.5.15.188:45483)
0926 14:53:55.850299 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.850301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.850308 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:53:55.850326 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.850327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.850336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.850392 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.850686 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.851407 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.852342 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.852535 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.852586 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.854309 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.854310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.854311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.854317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.855649 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.855981 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.855982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.855982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.856053 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.856275 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.856276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.856276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.856377 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.856445 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.856608 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.856672 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:53:55.856761 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.850034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51626)
0926 14:53:55.850124 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.850134 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.850168 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.850413 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.850415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.850562 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.850599 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.850601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.851493 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.851494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.852270 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.852272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.852607 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.852608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.852666 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.852969 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.852974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.854224 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.855732 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.855733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.855733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.855897 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.856121 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.856121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.856122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.856177 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.856450 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.856451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.856499 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.856582 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.856672 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 14:53:55.858604 17938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: Bootstrap starting.
I20260926 14:53:55.859270 17938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.859570 17938 log.cc:824] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.860307 17938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: No bootstrap required, opened a new log
I20260926 14:53:55.861903 17938 raft_consensus.cc:348] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.862035 17938 raft_consensus.cc:374] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.862087 17938 raft_consensus.cc:729] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d768e828db1a49e88b404bdffecb5842, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.862216 17938 consensus_queue.cc:260] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.862520 18115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.862668 18115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:53:55.862565 17938 sys_catalog.cc:564] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:53:55.864528 18126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:53:55.864612 18126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:53:55.865334 17870 master_runner.cc:186] Error getting master registration for 127.5.15.189:39715: OK, instance_id { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" instance_seqno: 1790434435656588 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:53:55.867636 17870 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:53:55.891237 18013 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } attempt: 3
Sep 26 14:53:55 dist-test-slave-s4p1 krb5kdc[17857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434435, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:53:55.899732 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.891642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.5.15.188:45483)
0926 14:53:55.891700 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:55.891703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:55.891733 (+ 30us) server_negotiation.cc:407] Connection header received
0926 14:53:55.891760 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:55.891761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:55.891772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:55.891830 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:55.892205 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.892975 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.894023 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.894333 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.894390 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.896852 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:55.896853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:55.896854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:55.896862 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:55.898487 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.898907 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.898908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.898908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.898968 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:55.899179 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:55.899180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:55.899180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:55.899318 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:55.899396 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:55.899567 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:53:55.899643 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:53:55.899734 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:55.891559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:45483 (local address 127.0.0.1:51642)
0926 14:53:55.891624 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:55.891637 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:55.891677 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:55.891854 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:55.891855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:55.892063 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:55.892119 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.892121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.893079 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.893080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.893922 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:55.893925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:55.894446 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:55.894447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:55.894502 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:55.895080 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:55.895086 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:55.896764 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:55.898575 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.898577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.898577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.898838 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.899040 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:55.899041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:55.899041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:55.899108 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:55.899399 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:55.899400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:55.899448 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 14:53:55.899527 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:53:55.899629 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 14:53:55.901695 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: Bootstrap starting.
I20260926 14:53:55.902433 18013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:55.902720 18013 log.cc:824] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:55.903976 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: No bootstrap required, opened a new log
I20260926 14:53:55.906064 18013 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.906204 18013 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:55.906247 18013 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad63ac7210a64748a112cd088640ed36, State: Initialized, Role: FOLLOWER
I20260926 14:53:55.906368 18013 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.906693 18013 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:55.906670 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.906883 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:53:55.908653 18140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:53:55.908752 18140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:53:55.910732 17945 master_runner.cc:263] No leader master found from master ad63ac7210a64748a112cd088640ed36
I20260926 14:53:55.910808 17945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:53:55.982689 18094 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:53:55.982856 18094 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.983137 18094 leader_election.cc:288] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d768e828db1a49e88b404bdffecb5842 (127.5.15.190:41583), ad63ac7210a64748a112cd088640ed36 (127.5.15.189:39715)
I20260926 14:53:55.983420 17912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a48aeb4b18d0449fbffad404cb4e4048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d768e828db1a49e88b404bdffecb5842" is_pre_election: true
I20260926 14:53:55.983464 17987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a48aeb4b18d0449fbffad404cb4e4048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad63ac7210a64748a112cd088640ed36" is_pre_election: true
I20260926 14:53:55.983583 17912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48aeb4b18d0449fbffad404cb4e4048 in term 0.
I20260926 14:53:55.983603 17987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48aeb4b18d0449fbffad404cb4e4048 in term 0.
I20260926 14:53:55.983788 18038 leader_election.cc:302] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a48aeb4b18d0449fbffad404cb4e4048, ad63ac7210a64748a112cd088640ed36; no voters:
I20260926 14:53:55.983938 18094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:53:55.983980 18094 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:53:55.984001 18094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.984690 18094 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.984817 18094 leader_election.cc:288] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [CANDIDATE]: Term 1 election: Requested vote from peers d768e828db1a49e88b404bdffecb5842 (127.5.15.190:41583), ad63ac7210a64748a112cd088640ed36 (127.5.15.189:39715)
I20260926 14:53:55.984937 17912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a48aeb4b18d0449fbffad404cb4e4048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d768e828db1a49e88b404bdffecb5842"
I20260926 14:53:55.985013 17987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a48aeb4b18d0449fbffad404cb4e4048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad63ac7210a64748a112cd088640ed36"
I20260926 14:53:55.985015 17912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.985074 17987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:55.985890 17987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48aeb4b18d0449fbffad404cb4e4048 in term 1.
I20260926 14:53:55.985890 17912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48aeb4b18d0449fbffad404cb4e4048 in term 1.
I20260926 14:53:55.986060 18038 leader_election.cc:302] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a48aeb4b18d0449fbffad404cb4e4048, ad63ac7210a64748a112cd088640ed36; no voters:
I20260926 14:53:55.986157 18094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:55.986339 18094 raft_consensus.cc:686] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [term 1 LEADER]: Becoming Leader. State: Replica: a48aeb4b18d0449fbffad404cb4e4048, State: Running, Role: LEADER
I20260926 14:53:55.986449 18094 consensus_queue.cc:237] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } }
I20260926 14:53:55.986853 18094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48aeb4b18d0449fbffad404cb4e4048. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.986922 18094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:55.987327 18144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:55.987535 18144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:55.988765 17987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [term 1 FOLLOWER]: Refusing update from remote peer a48aeb4b18d0449fbffad404cb4e4048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:55.988840 17912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [term 1 FOLLOWER]: Refusing update from remote peer a48aeb4b18d0449fbffad404cb4e4048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:53:55.989073 18142 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:55.989166 18094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:53:55.990329 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48aeb4b18d0449fbffad404cb4e4048. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.990312 18115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48aeb4b18d0449fbffad404cb4e4048. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.990397 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:53:55.990396 18115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:53:55.991331 18094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.991413 18094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:55.991672 18145 mvcc.cc:205] Tried to move back new op lower bound from 7333619449808125952 to 7333619449801089024. Current Snapshot: MvccSnapshot[applied={T|T < 7333619449808125952}]
I20260926 14:53:55.991809 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.991880 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048 [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:55.992093 18115 sys_catalog.cc:455] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.992178 18115 sys_catalog.cc:458] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:53:55.992359 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a48aeb4b18d0449fbffad404cb4e4048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d768e828db1a49e88b404bdffecb5842" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 41583 } } peers { permanent_uuid: "ad63ac7210a64748a112cd088640ed36" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39715 } } peers { permanent_uuid: "a48aeb4b18d0449fbffad404cb4e4048" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 45483 } } }
I20260926 14:53:55.992434 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:53:55.992584 18146 mvcc.cc:205] Tried to move back new op lower bound from 7333619449808125952 to 7333619449801089024. Current Snapshot: MvccSnapshot[applied={T|T < 7333619449808125952}]
I20260926 14:53:55.993098 18147 mvcc.cc:205] Tried to move back new op lower bound from 7333619449808125952 to 7333619449801089024. Current Snapshot: MvccSnapshot[applied={T|T < 7333619449808125952}]
I20260926 14:53:55.993834 18144 catalog_manager.cc:1347] Generated new cluster ID: 52b50d96dc42443e874d15fc9a82688b
I20260926 14:53:55.993909 18144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:56.006034 18144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:56.006714 18144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:56.013355 18144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a48aeb4b18d0449fbffad404cb4e4048: Generated new TSK 0
I20260926 14:53:56.013553 18144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:53:56.865028 18126 catalog_manager.cc:1259] Loaded cluster ID: 52b50d96dc42443e874d15fc9a82688b
I20260926 14:53:56.865096 18126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: loading cluster ID for follower catalog manager: success
I20260926 14:53:56.865936 18126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: acquiring CA information for follower catalog manager: success
I20260926 14:53:56.866190 18126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d768e828db1a49e88b404bdffecb5842: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:53:56.909173 18140 catalog_manager.cc:1259] Loaded cluster ID: 52b50d96dc42443e874d15fc9a82688b
I20260926 14:53:56.909242 18140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: loading cluster ID for follower catalog manager: success
I20260926 14:53:56.910002 18140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: acquiring CA information for follower catalog manager: success
I20260926 14:53:56.910256 18140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad63ac7210a64748a112cd088640ed36: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:53:57.062798 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17870
I20260926 14:53:57.067787 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 17945
I20260926 14:53:57.072589 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18020
2026-09-26T14:53:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2816 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:57 dist-test-slave-s4p1 krb5kdc[18158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:57 dist-test-slave-s4p1 krb5kdc[18158](info): set up 2 sockets
Sep 26 14:53:57 dist-test-slave-s4p1 krb5kdc[18158](info): commencing operation
krb5kdc: starting...
W20260926 14:53:58.095093 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:53:58 dist-test-slave-s4p1 krb5kdc[18158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:53:58.203644 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:42819
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:39047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:58.281040 18171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:58.281220 18171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:58.281240 18171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:58.282234 18171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:58.282266 18171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:58.282280 18171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:58.282292 18171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:58.282303 18171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:58.283674 18171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:39047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:42819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:58.283842 18171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:58.283991 18171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:53:58.288509 18171 server_base.cc:1034] running on GCE node
W20260926 14:53:58.288436 18176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.288435 18179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:58.288468 18177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:58.288873 18171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:58.289155 18171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:58.290313 18171 hybrid_clock.cc:648] HybridClock initialized: now 1790434438290309 us; error 34 us; skew 500 ppm
Sep 26 14:53:58 dist-test-slave-s4p1 krb5kdc[18158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434438, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:58.292624 18171 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:53:58.293068 18171 webserver.cc:460] Webserver started at http://127.5.15.190:39533/ using document root <none> and password file <none>
I20260926 14:53:58.293298 18171 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:58.293365 18171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:58.293514 18171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:58.294476 18171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "b66b89094e8f427a9f4d7892308c279b"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-s4p1"
server_key: "b77ca852ba69cbc7b7110da0a2ac77f6"
server_key_iv: "c8f3c20fa7e580f71ba75899fbe61041"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.294876 18171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "b66b89094e8f427a9f4d7892308c279b"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-s4p1"
server_key: "b77ca852ba69cbc7b7110da0a2ac77f6"
server_key_iv: "c8f3c20fa7e580f71ba75899fbe61041"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.296213 18171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:53:58.296885 18186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:58.297044 18171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:53:58.297138 18171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b66b89094e8f427a9f4d7892308c279b"
format_stamp: "Formatted at 2026-09-26 14:53:58 on dist-test-slave-s4p1"
server_key: "b77ca852ba69cbc7b7110da0a2ac77f6"
server_key_iv: "c8f3c20fa7e580f71ba75899fbe61041"
server_key_version: "encryptionkey@0"
I20260926 14:53:58.297219 18171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:58.313297 18171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:58.315536 18171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:58.315724 18171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:58.319922 18171 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:42819
I20260926 14:53:58.319968 18238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:42819 every 8 connection(s)
I20260926 14:53:58.320297 18171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:58.321086 18239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:58.323326 18239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b: Bootstrap starting.
I20260926 14:53:58.323845 18239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:58.324175 18239 log.cc:824] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:58.327453 18239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b: No bootstrap required, opened a new log
I20260926 14:53:58.327642 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18171
I20260926 14:53:58.327733 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:53:58.327903 5182 external_mini_cluster.cc:1428] Setting key 9d5682789043e1ed9d3b278a88865ddc
I20260926 14:53:58.330001 18239 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b89094e8f427a9f4d7892308c279b" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42819 } }
I20260926 14:53:58.330196 18239 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:58.330246 18239 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66b89094e8f427a9f4d7892308c279b, State: Initialized, Role: FOLLOWER
I20260926 14:53:58.330359 18239 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b89094e8f427a9f4d7892308c279b" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42819 } }
I20260926 14:53:58.330436 18239 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:58.330466 18239 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:58.330508 18239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:58.331144 18239 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b89094e8f427a9f4d7892308c279b" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42819 } }
I20260926 14:53:58.331272 18239 leader_election.cc:302] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66b89094e8f427a9f4d7892308c279b; no voters:
I20260926 14:53:58.331429 18239 leader_election.cc:288] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:58.331506 18244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:58.331655 18244 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [term 1 LEADER]: Becoming Leader. State: Replica: b66b89094e8f427a9f4d7892308c279b, State: Running, Role: LEADER
I20260926 14:53:58.331717 18239 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:58.331794 18244 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b89094e8f427a9f4d7892308c279b" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42819 } }
I20260926 14:53:58.332813 18246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66b89094e8f427a9f4d7892308c279b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b89094e8f427a9f4d7892308c279b" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42819 } } }
I20260926 14:53:58.332912 18246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:58.333051 18245 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66b89094e8f427a9f4d7892308c279b. Latest consensus state: current_term: 1 leader_uuid: "b66b89094e8f427a9f4d7892308c279b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b89094e8f427a9f4d7892308c279b" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 42819 } } }
I20260926 14:53:58.333123 18245 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:58.333353 18259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:58.333747 18259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:58.335457 18259 catalog_manager.cc:1347] Generated new cluster ID: 83bc18136a7a49f9ae96080d707bb45f
I20260926 14:53:58.335510 18259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:53:58 dist-test-slave-s4p1 krb5kdc[18158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:58.340133 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:58.328429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.5.15.190:42819)
0926 14:53:58.328639 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:58.328648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:58.328713 (+ 65us) server_negotiation.cc:407] Connection header received
0926 14:53:58.330245 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:58.330254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:58.330316 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:58.330458 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:58.331230 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.332678 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.333793 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:58.334362 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:58.334470 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:58.336880 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:58.336884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:58.336884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:58.336896 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:58.338535 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:58.338987 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:58.338990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:58.338990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:58.339078 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:58.339316 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:58.339316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:58.339317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:58.339540 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:58.339741 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:58.339847 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 14:53:58.339985 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
I20260926 14:53:58.341041 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:53:58.346477 18259 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:58.347141 18259 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:58.356535 18259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66b89094e8f427a9f4d7892308c279b: Generated new TSK 0
I20260926 14:53:58.356722 18259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:53:58.394922 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18171
2026-09-26T14:53:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1320 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:53:58 dist-test-slave-s4p1 krb5kdc[18267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:53:58 dist-test-slave-s4p1 krb5kdc[18267](info): set up 2 sockets
Sep 26 14:53:58 dist-test-slave-s4p1 krb5kdc[18267](info): commencing operation
krb5kdc: starting...
W20260926 14:53:59.418807 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:53:59.527451 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:45943
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:59.605677 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:59.605863 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:59.605885 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:59.606972 18280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:53:59.607014 18280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:59.607028 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:59.607040 18280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:53:59.607051 18280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:53:59.608366 18280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:45943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:59.608536 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:59.608683 18280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:59.613308 18285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.613435 18280 server_base.cc:1034] running on GCE node
W20260926 14:53:59.613301 18286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:59.613387 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.613716 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:59.613996 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:59.615135 18280 hybrid_clock.cc:648] HybridClock initialized: now 1790434439615124 us; error 29 us; skew 500 ppm
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:59.617440 18280 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:53:59.617852 18280 webserver.cc:460] Webserver started at http://127.5.15.190:36489/ using document root <none> and password file <none>
I20260926 14:53:59.618052 18280 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:59.618090 18280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:59.618171 18280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:59.619179 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "07a2f401097e4b10b5596109b153d2bf"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "9ab6fc18e2bf73cc9cca2c1a9ca0f608"
server_key_iv: "7e6e6cc0b76f00eb9bf01f074882b02c"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.619473 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "07a2f401097e4b10b5596109b153d2bf"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "9ab6fc18e2bf73cc9cca2c1a9ca0f608"
server_key_iv: "7e6e6cc0b76f00eb9bf01f074882b02c"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.620718 18280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:53:59.621389 18295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.621662 18280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:53:59.621764 18280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "07a2f401097e4b10b5596109b153d2bf"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "9ab6fc18e2bf73cc9cca2c1a9ca0f608"
server_key_iv: "7e6e6cc0b76f00eb9bf01f074882b02c"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.621838 18280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:59.633682 18280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:59.635895 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:59.636051 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:59.639971 18280 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:45943
I20260926 14:53:59.640000 18347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:45943 every 8 connection(s)
I20260926 14:53:59.640345 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:53:59.641116 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18280
I20260926 14:53:59.641209 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:53:59.641394 5182 external_mini_cluster.cc:1428] Setting key b09cd632c89559e6b6e00630b68adc22
I20260926 14:53:59.641470 18348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:53:59.644775 18348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap starting.
I20260926 14:53:59.645346 18348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Neither blocks nor log segments found. Creating new log.
I20260926 14:53:59.645661 18348 log.cc:824] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Log is configured to *not* fsync() on all Append() calls
I20260926 14:53:59.646348 18348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: No bootstrap required, opened a new log
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:59.648267 18348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:53:59.648411 18348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:53:59.648502 18348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a2f401097e4b10b5596109b153d2bf, State: Initialized, Role: FOLLOWER
I20260926 14:53:59.648658 18348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:53:59.648794 18348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:53:59.648830 18348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:53:59.648890 18348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:53:59.649490 18348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:53:59.649626 18348 leader_election.cc:302] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a2f401097e4b10b5596109b153d2bf; no voters:
I20260926 14:53:59.649803 18348 leader_election.cc:288] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:53:59.649843 18353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:53:59.650020 18353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 LEADER]: Becoming Leader. State: Replica: 07a2f401097e4b10b5596109b153d2bf, State: Running, Role: LEADER
I20260926 14:53:59.650061 18348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:53:59.650178 18353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:53:59.650604 18355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a2f401097e4b10b5596109b153d2bf. Latest consensus state: current_term: 1 leader_uuid: "07a2f401097e4b10b5596109b153d2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } } }
I20260926 14:53:59.650698 18355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:59.650938 18354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a2f401097e4b10b5596109b153d2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } } }
I20260926 14:53:59.651262 18362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:53:59.651312 18354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: This master's current role is: LEADER
I20260926 14:53:59.651835 18362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:53:59.652323 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.641870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.5.15.190:45943)
0926 14:53:59.642067 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.642077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.642113 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:53:59.642642 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.642650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.642708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.642870 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:59.643411 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.644756 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.645517 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.646083 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.646190 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.648489 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:59.648495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:59.648495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:59.648510 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:59.650510 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:59.650981 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:59.650984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:59.650984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:59.651084 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:59.651312 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:59.651313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:59.651313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:59.651674 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:59.651816 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.651998 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.652134 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260926 14:53:59.653255 18362 catalog_manager.cc:1347] Generated new cluster ID: fd2cb1741d6347adaf9e65ab4c5880c5
I20260926 14:53:59.653335 18362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:53:59.665822 18362 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:53:59.666298 18362 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:53:59.669661 18362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Generated new TSK 0
I20260926 14:53:59.669831 18362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:53:59.702471 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:45943
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:59.780637 18374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:59.780822 18374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:59.780844 18374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:59.781898 18374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:59.781930 18374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:59.781955 18374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:53:59.783274 18374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:59.783432 18374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:59.783591 18374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:59.788228 18379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:59.788235 18382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:59.788235 18380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.788307 18374 server_base.cc:1034] running on GCE node
I20260926 14:53:59.788587 18374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:59.788805 18374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:59.789971 18374 hybrid_clock.cc:648] HybridClock initialized: now 1790434439789959 us; error 24 us; skew 500 ppm
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:59.792413 18374 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:53:59.792801 18374 webserver.cc:460] Webserver started at http://127.5.15.129:35759/ using document root <none> and password file <none>
I20260926 14:53:59.793005 18374 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:59.793075 18374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:59.793216 18374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:59.794159 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "0c62b76e99be4851ba43d08c5b5d4d16"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "6b84ec8cfd76819a809de8bc0a10727c"
server_key_iv: "7ebfdcbddffff2b75737b443c1867872"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.794500 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "0c62b76e99be4851ba43d08c5b5d4d16"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "6b84ec8cfd76819a809de8bc0a10727c"
server_key_iv: "7ebfdcbddffff2b75737b443c1867872"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.795768 18374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:53:59.796490 18389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.796660 18374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:53:59.796751 18374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "0c62b76e99be4851ba43d08c5b5d4d16"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "6b84ec8cfd76819a809de8bc0a10727c"
server_key_iv: "7ebfdcbddffff2b75737b443c1867872"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.796830 18374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:59.803341 18374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:59.805152 18374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:59.805294 18374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:59.805517 18374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:59.805804 18374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:59.805856 18374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.805896 18374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:59.805924 18374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.811619 18374 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:44297
I20260926 14:53:59.811689 18502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:44297 every 8 connection(s)
I20260926 14:53:59.811962 18374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:53:59.816587 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18374
I20260926 14:53:59.816660 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:53:59.816850 5182 external_mini_cluster.cc:1428] Setting key 41aec6a6d75cabb0aab7c296203a5856
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:59.820806 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.812619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:49215)
0926 14:53:59.812763 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.812793 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.813203 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.813380 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.813383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.813877 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:59.814068 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.814072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.814742 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.814742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.815546 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.815548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.815879 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.815880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.815967 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.816352 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:59.816359 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:59.817863 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:59.819514 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:59.819524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:59.819526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:59.819748 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:59.819983 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:59.819984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:59.819985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:59.820082 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:59.820373 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:59.820378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:59.820476 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.820561 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.820667 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260926 14:53:59.820832 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.812660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:49215 (local address 127.5.15.190:45943)
0926 14:53:59.812740 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.812742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.812868 (+ 126us) server_negotiation.cc:407] Connection header received
0926 14:53:59.813277 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.813279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.813288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.813337 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:59.814148 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.814665 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.815616 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.815810 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.815862 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.817950 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:59.817951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:59.817951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:59.817971 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:59.819428 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:59.819848 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:59.819849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:59.819849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:59.819909 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:59.820153 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:59.820154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:59.820154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:59.820308 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:59.820378 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.820607 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.820695 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:53:59.821475 18503 heartbeater.cc:347] Connected to a master server at 127.5.15.190:45943
I20260926 14:53:59.821574 18503 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:59.821724 18503 heartbeater.cc:511] Master 127.5.15.190:45943 requested a full tablet report, sending...
I20260926 14:53:59.822238 18312 ts_manager.cc:194] Registered new tserver with Master: 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297)
I20260926 14:53:59.822903 18312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:49215
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:53:59.841737 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:45943
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:53:59.921135 18508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:53:59.921324 18508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:53:59.921347 18508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:53:59.922590 18508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:53:59.922633 18508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:53:59.922659 18508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:53:59.924180 18508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:53:59.924436 18508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:53:59.924690 18508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:53:59.929476 18513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.929661 18508 server_base.cc:1034] running on GCE node
W20260926 14:53:59.929489 18516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:53:59.929486 18514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:53:59.929976 18508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:53:59.930235 18508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:53:59.931391 18508 hybrid_clock.cc:648] HybridClock initialized: now 1790434439931371 us; error 32 us; skew 500 ppm
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:53:59.933769 18508 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:53:59.934161 18508 webserver.cc:460] Webserver started at http://127.5.15.130:33797/ using document root <none> and password file <none>
I20260926 14:53:59.934376 18508 fs_manager.cc:359] Metadata directory not provided
I20260926 14:53:59.934441 18508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:53:59.934593 18508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:53:59.935614 18508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "f8704464d96d4fedb16a32949e12f458"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "8d755bdf0a5b1c9d6463a235fe77f81f"
server_key_iv: "b883530b1f9f36031b754c910681b1f6"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.935961 18508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "f8704464d96d4fedb16a32949e12f458"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "8d755bdf0a5b1c9d6463a235fe77f81f"
server_key_iv: "b883530b1f9f36031b754c910681b1f6"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.937211 18508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:53:59.937871 18523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.938082 18508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:53:59.938176 18508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "f8704464d96d4fedb16a32949e12f458"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "8d755bdf0a5b1c9d6463a235fe77f81f"
server_key_iv: "b883530b1f9f36031b754c910681b1f6"
server_key_version: "encryptionkey@0"
I20260926 14:53:59.938259 18508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:53:59.948719 18508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:53:59.951083 18508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:53:59.951234 18508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:53:59.951452 18508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:53:59.951745 18508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:53:59.951804 18508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.951844 18508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:53:59.951876 18508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:53:59.957362 18508 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:37113
I20260926 14:53:59.957397 18636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:37113 every 8 connection(s)
I20260926 14:53:59.957707 18508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:53:59 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434439, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:53:59.965869 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.958412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:59573 (local address 127.5.15.190:45943)
0926 14:53:59.958475 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:53:59.958477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:53:59.958669 (+ 192us) server_negotiation.cc:407] Connection header received
0926 14:53:59.959192 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:53:59.959193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:53:59.959208 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:53:59.959267 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:53:59.960106 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.960589 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.961510 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.961693 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.961742 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.963633 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:53:59.963634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:53:59.963634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:53:59.963640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:53:59.964763 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:59.965115 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:59.965116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:59.965116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:59.965163 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:53:59.965388 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:53:59.965388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:53:59.965389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:53:59.965484 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:53:59.965540 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:53:59.965750 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 14:53:59.965807 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:53:59.966003 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18508
I20260926 14:53:59.966104 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:53:59.966121 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:53:59.958387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:59573)
0926 14:53:59.958547 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 14:53:59.958586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:53:59.959091 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:53:59.959325 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:53:59.959328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:53:59.959850 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:53:59.960019 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.960023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.960665 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.960666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.961435 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:53:59.961437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:53:59.961762 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:53:59.961763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:53:59.961883 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:53:59.962272 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:53:59.962279 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:53:59.963553 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:53:59.964835 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:59.964841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:59.964843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:59.965051 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:59.965227 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:53:59.965227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:53:59.965227 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:53:59.965318 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:53:59.965539 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:53:59.965543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:53:59.965646 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 14:53:59.965725 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:53:59.965881 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260926 14:53:59.966321 5182 external_mini_cluster.cc:1428] Setting key a75f71f5207136b74e49881fd45dd235
I20260926 14:53:59.966637 18637 heartbeater.cc:347] Connected to a master server at 127.5.15.190:45943
I20260926 14:53:59.966749 18637 heartbeater.cc:464] Registering TS with master...
I20260926 14:53:59.966941 18637 heartbeater.cc:511] Master 127.5.15.190:45943 requested a full tablet report, sending...
I20260926 14:53:59.967335 18312 ts_manager.cc:194] Registered new tserver with Master: f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
I20260926 14:53:59.967706 18312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:59573
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:53:59.989579 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:45943
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:00.068065 18642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.068280 18642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.068303 18642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.069411 18642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.069451 18642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.069476 18642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:54:00.070940 18642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:00.071135 18642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.071429 18642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.076289 18648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.076299 18650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.076300 18647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.076395 18642 server_base.cc:1034] running on GCE node
I20260926 14:54:00.076713 18642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.076967 18642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.078107 18642 hybrid_clock.cc:648] HybridClock initialized: now 1790434440078098 us; error 30 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.080605 18642 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:54:00.080997 18642 webserver.cc:460] Webserver started at http://127.5.15.131:43975/ using document root <none> and password file <none>
I20260926 14:54:00.081167 18642 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.081209 18642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.081295 18642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:00.082265 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "a405277676a34f93ad239d2059f7b9e0"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-s4p1"
server_key: "f17383f27752fd930ebc06be63cd9a3a"
server_key_iv: "05e11d947b03d15ba00e59e8ba7f5bb8"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.082551 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "a405277676a34f93ad239d2059f7b9e0"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-s4p1"
server_key: "f17383f27752fd930ebc06be63cd9a3a"
server_key_iv: "05e11d947b03d15ba00e59e8ba7f5bb8"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.083912 18642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:54:00.084599 18657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.084790 18642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:54:00.084861 18642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "a405277676a34f93ad239d2059f7b9e0"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-s4p1"
server_key: "f17383f27752fd930ebc06be63cd9a3a"
server_key_iv: "05e11d947b03d15ba00e59e8ba7f5bb8"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.084914 18642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.097673 18642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.100023 18642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.100135 18642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.100311 18642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:00.100615 18642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:00.100649 18642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.100694 18642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:00.100754 18642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.106557 18642 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46677
I20260926 14:54:00.106638 18770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46677 every 8 connection(s)
I20260926 14:54:00.106961 18642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:00.113924 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18642
I20260926 14:54:00.114027 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:54:00.114214 5182 external_mini_cluster.cc:1428] Setting key db59a9d85d78d7b924962c9449e7b010
I20260926 14:54:00.116060 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.107711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:46431 (local address 127.5.15.190:45943)
0926 14:54:00.107757 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.107759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.107952 (+ 193us) server_negotiation.cc:407] Connection header received
0926 14:54:00.108432 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.108433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.108444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.108496 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.109383 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.109883 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.110893 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.111077 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.111129 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.113190 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.113191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.113191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.113197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.114799 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.115255 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.115256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.115256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.115324 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.115536 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.115536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.115537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.115637 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.115687 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.115935 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.115995 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:54:00.116255 18773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.107651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:46431)
0926 14:54:00.107828 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.107863 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.108348 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.108549 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.108553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.109105 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.109305 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.109310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.109959 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.109960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.110809 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.110812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.111147 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.111148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.111255 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.111697 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.111707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.113101 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.114875 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.114882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.114884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.115190 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.115381 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.115381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.115381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.115476 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.115693 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.115705 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.115819 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.115912 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.116024 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260926 14:54:00.116740 18771 heartbeater.cc:347] Connected to a master server at 127.5.15.190:45943
I20260926 14:54:00.116817 18771 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:00.117007 18771 heartbeater.cc:511] Master 127.5.15.190:45943 requested a full tablet report, sending...
I20260926 14:54:00.117376 18312 ts_manager.cc:194] Registered new tserver with Master: a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677)
I20260926 14:54:00.117702 18312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:46431
I20260926 14:54:00.126119 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:54:00.132941 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.127494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.5.15.190:45943)
0926 14:54:00.127531 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.127533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.127644 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:54:00.127681 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.127682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.127690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.127736 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.128321 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.128799 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.129504 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.129677 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.129724 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.130877 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.130878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.130878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.130884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.132037 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.132337 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.132337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.132338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.132384 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.132540 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.132540 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.132540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.132660 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.132733 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.132819 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.132878 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:54:00.134557 18312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:54:00.136198 18312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:54:00.142539 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.138488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.5.15.129:44297)
0926 14:54:00.139020 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.139023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.139036 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:00.139097 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.139099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.139113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.139196 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.139293 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.140310 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.141456 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.142126 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.142211 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.142271 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.142422 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.142460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":75,"threads_started":1}
I20260926 14:54:00.142540 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.138245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44297 (local address 127.0.0.1:47474)
0926 14:54:00.138633 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.138641 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.138681 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.139193 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.139194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.139196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.139235 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.139237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.140408 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.140409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.141372 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.141374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.142211 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.142211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.142286 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.142301 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.142402 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.142457 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":1,"thread_start_us":52,"threads_started":1}
I20260926 14:54:00.142795 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.138351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.5.15.130:37113)
0926 14:54:00.138690 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.138694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.138704 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:54:00.138769 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.138771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.138789 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.138886 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.139255 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.139958 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.141496 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.142249 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.142320 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.142373 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.142642 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.142682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":75,"threads_started":1}
I20260926 14:54:00.142992 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.138475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.5.15.131:46677)
0926 14:54:00.138686 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.138692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.138703 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:54:00.138773 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.138775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.138789 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.138881 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.139084 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.140196 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.141521 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.142255 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.142320 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.142373 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.142869 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.142910 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":110,"threads_started":1}
I20260926 14:54:00.143239 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.138244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:37113 (local address 127.0.0.1:38300)
0926 14:54:00.138506 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.138517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.138558 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.139127 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.139128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.139130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.139162 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.139163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.140061 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.140063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.141396 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.141399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.142412 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.142413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.142481 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.142490 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.142540 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.142597 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":9,"thread_start_us":104,"threads_started":1}
I20260926 14:54:00.143409 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.138243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46677 (local address 127.0.0.1:40134)
0926 14:54:00.138420 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.138462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.138532 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.138931 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.138934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.138937 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.139005 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.139007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.140296 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.140297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.141440 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.141441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.142411 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.142413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.142481 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.142492 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.142540 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.142597 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260926 14:54:00.143435 18437 tablet_service.cc:1467] Processing CreateTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f (DEFAULT_TABLE table=test-table [id=c459a7d1aa9649cfb8d5d47376e136c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:00.143849 18437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09a9b1f9d044374beb7a9a1e943cc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.144098 18705 tablet_service.cc:1467] Processing CreateTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f (DEFAULT_TABLE table=test-table [id=c459a7d1aa9649cfb8d5d47376e136c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:00.144161 18571 tablet_service.cc:1467] Processing CreateTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f (DEFAULT_TABLE table=test-table [id=c459a7d1aa9649cfb8d5d47376e136c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:00.144366 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09a9b1f9d044374beb7a9a1e943cc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.144529 18571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b09a9b1f9d044374beb7a9a1e943cc0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.146864 18785 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Bootstrap starting.
I20260926 14:54:00.147336 18786 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap starting.
I20260926 14:54:00.147670 18785 tablet_bootstrap.cc:654] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.147928 18786 tablet_bootstrap.cc:654] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.147980 18787 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Bootstrap starting.
I20260926 14:54:00.148247 18785 log.cc:824] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.148553 18786 log.cc:824] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.148599 18787 tablet_bootstrap.cc:654] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.149052 18785 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: No bootstrap required, opened a new log
I20260926 14:54:00.149127 18785 ts_tablet_manager.cc:1397] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:00.149173 18787 log.cc:824] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.149333 18786 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: No bootstrap required, opened a new log
I20260926 14:54:00.149467 18786 ts_tablet_manager.cc:1397] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:00.151338 18785 raft_consensus.cc:348] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.151388 18786 raft_consensus.cc:348] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.151535 18785 raft_consensus.cc:374] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.151552 18786 raft_consensus.cc:374] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.151578 18785 raft_consensus.cc:729] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405277676a34f93ad239d2059f7b9e0, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.151587 18786 raft_consensus.cc:729] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c62b76e99be4851ba43d08c5b5d4d16, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.151710 18786 consensus_queue.cc:260] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.151700 18785 consensus_queue.cc:260] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.152004 18786 ts_tablet_manager.cc:1428] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:00.152088 18785 ts_tablet_manager.cc:1428] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:54:00.152012 18503 heartbeater.cc:503] Master 127.5.15.190:45943 was elected leader, sending a full tablet report...
I20260926 14:54:00.152102 18771 heartbeater.cc:503] Master 127.5.15.190:45943 was elected leader, sending a full tablet report...
I20260926 14:54:00.153020 18787 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: No bootstrap required, opened a new log
I20260926 14:54:00.153090 18787 ts_tablet_manager.cc:1397] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 14:54:00.154613 18787 raft_consensus.cc:348] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.154735 18787 raft_consensus.cc:374] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.154814 18787 raft_consensus.cc:729] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8704464d96d4fedb16a32949e12f458, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.154925 18787 consensus_queue.cc:260] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.155208 18637 heartbeater.cc:503] Master 127.5.15.190:45943 was elected leader, sending a full tablet report...
I20260926 14:54:00.155215 18787 ts_tablet_manager.cc:1428] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260926 14:54:00.208272 18638 tablet.cc:2369] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:00.275558 18793 raft_consensus.cc:482] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:00.275676 18793 raft_consensus.cc:504] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.276098 18793 leader_election.cc:288] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297)
I20260926 14:54:00.279990 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.276355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:60969 (local address 127.5.15.129:44297)
0926 14:54:00.276416 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.276418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.276471 (+ 53us) server_negotiation.cc:407] Connection header received
0926 14:54:00.276501 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.276502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.276515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.276567 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.276759 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.277525 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.278995 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.279628 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.279715 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.279722 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.279884 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.279922 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:00.279990 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.276339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:54593 (local address 127.5.15.131:46677)
0926 14:54:00.276382 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.276384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.276619 (+ 235us) server_negotiation.cc:407] Connection header received
0926 14:54:00.276686 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.276687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.276694 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.276767 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.276899 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.277582 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.278990 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.279628 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.279709 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.279718 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.279884 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.279922 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:54:00.280018 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.276296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44297 (local address 127.5.15.130:60969)
0926 14:54:00.276346 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.276358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.276407 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.276600 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.276600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.276602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.276676 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.276679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.277601 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.277601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.278906 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.278908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.279714 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.279714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.279797 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.279799 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.279860 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.279910 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":33}
I20260926 14:54:00.280025 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.276295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46677 (local address 127.5.15.130:54593)
0926 14:54:00.276546 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.276572 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.276626 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.276779 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.276780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.276781 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.276837 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.276838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.277653 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.277653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.278906 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.278908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.279714 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.279715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.279793 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.279797 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.279860 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.279909 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":97,"thread_start_us":115,"threads_started":1}
I20260926 14:54:00.280427 18457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09a9b1f9d044374beb7a9a1e943cc0f" candidate_uuid: "f8704464d96d4fedb16a32949e12f458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" is_pre_election: true
I20260926 14:54:00.280447 18725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09a9b1f9d044374beb7a9a1e943cc0f" candidate_uuid: "f8704464d96d4fedb16a32949e12f458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405277676a34f93ad239d2059f7b9e0" is_pre_election: true
I20260926 14:54:00.280584 18457 raft_consensus.cc:2454] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8704464d96d4fedb16a32949e12f458 in term 0.
I20260926 14:54:00.280596 18725 raft_consensus.cc:2454] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8704464d96d4fedb16a32949e12f458 in term 0.
I20260926 14:54:00.280805 18527 leader_election.cc:302] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a405277676a34f93ad239d2059f7b9e0, f8704464d96d4fedb16a32949e12f458; no voters:
I20260926 14:54:00.280961 18793 raft_consensus.cc:2781] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:54:00.281016 18793 raft_consensus.cc:482] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:54:00.281042 18793 raft_consensus.cc:3037] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:00.281906 18793 raft_consensus.cc:504] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.282037 18793 leader_election.cc:288] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [CANDIDATE]: Term 1 election: Requested vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297)
I20260926 14:54:00.282179 18457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09a9b1f9d044374beb7a9a1e943cc0f" candidate_uuid: "f8704464d96d4fedb16a32949e12f458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c62b76e99be4851ba43d08c5b5d4d16"
I20260926 14:54:00.282172 18725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b09a9b1f9d044374beb7a9a1e943cc0f" candidate_uuid: "f8704464d96d4fedb16a32949e12f458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405277676a34f93ad239d2059f7b9e0"
I20260926 14:54:00.282274 18457 raft_consensus.cc:3037] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:00.282270 18725 raft_consensus.cc:3037] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:00.283053 18457 raft_consensus.cc:2454] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8704464d96d4fedb16a32949e12f458 in term 1.
I20260926 14:54:00.283075 18725 raft_consensus.cc:2454] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8704464d96d4fedb16a32949e12f458 in term 1.
I20260926 14:54:00.283238 18525 leader_election.cc:302] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16, f8704464d96d4fedb16a32949e12f458; no voters:
I20260926 14:54:00.283322 18793 raft_consensus.cc:2781] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:00.283485 18793 raft_consensus.cc:686] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 1 LEADER]: Becoming Leader. State: Replica: f8704464d96d4fedb16a32949e12f458, State: Running, Role: LEADER
I20260926 14:54:00.283578 18793 consensus_queue.cc:237] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.284369 18311 catalog_manager.cc:5495] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8704464d96d4fedb16a32949e12f458 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "f8704464d96d4fedb16a32949e12f458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } health_report { overall_health: UNKNOWN } } }
W20260926 14:54:00.312673 18504 tablet.cc:2369] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:00.322332 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.319882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.5.15.130:37113)
0926 14:54:00.319922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.319924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.319933 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:54:00.319995 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.319996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.320025 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.320102 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:00.320254 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.320716 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.321504 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.321713 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.321792 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.321967 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:00.322070 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:00.322167 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.322187 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.322230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:54:00.324164 18725 raft_consensus.cc:1261] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Refusing update from remote peer f8704464d96d4fedb16a32949e12f458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:00.324200 18457 raft_consensus.cc:1261] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Refusing update from remote peer f8704464d96d4fedb16a32949e12f458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:00.324599 18795 consensus_queue.cc:1035] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:00.328003 18798 mvcc.cc:205] Tried to move back new op lower bound from 7333619467564892160 to 7333619467402043392. Current Snapshot: MvccSnapshot[applied={T|T < 7333619467564892160 or (T in {7333619467564892160})}]
I20260926 14:54:00.328137 18799 mvcc.cc:205] Tried to move back new op lower bound from 7333619467564892160 to 7333619467402043392. Current Snapshot: MvccSnapshot[applied={T|T < 7333619467564892160}]
I20260926 14:54:00.329986 18793 consensus_queue.cc:1035] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:00.331604 18311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:00.331713 18311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:00.334676 18311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Sending DeleteTablet for 3 replicas of tablet b09a9b1f9d044374beb7a9a1e943cc0f
I20260926 14:54:00.335052 18437 tablet_service.cc:1514] Processing DeleteTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:00 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:47474
I20260926 14:54:00.335100 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18374
I20260926 14:54:00.335177 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:00 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:40134
I20260926 14:54:00.335678 18571 tablet_service.cc:1514] Processing DeleteTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:00 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:38300
I20260926 14:54:00.335898 18807 tablet_replica.cc:330] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: stopping tablet replica
I20260926 14:54:00.335994 18807 raft_consensus.cc:2229] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:00.336114 18807 raft_consensus.cc:2258] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:00.336268 18808 tablet_replica.cc:330] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: stopping tablet replica
I20260926 14:54:00.336388 18808 raft_consensus.cc:2229] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:54:00.336532 18808 raft_consensus.cc:2258] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:00.336995 18808 ts_tablet_manager.cc:1905] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:00.337064 18807 ts_tablet_manager.cc:1905] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:00.339075 18807 ts_tablet_manager.cc:1918] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:00.339184 18807 log.cc:1196] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/b09a9b1f9d044374beb7a9a1e943cc0f
I20260926 14:54:00.339219 18808 ts_tablet_manager.cc:1918] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:00.339375 18808 log.cc:1196] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/b09a9b1f9d044374beb7a9a1e943cc0f
I20260926 14:54:00.339497 18807 ts_tablet_manager.cc:1939] T b09a9b1f9d044374beb7a9a1e943cc0f P a405277676a34f93ad239d2059f7b9e0: Deleting consensus metadata
I20260926 14:54:00.339725 18808 ts_tablet_manager.cc:1939] T b09a9b1f9d044374beb7a9a1e943cc0f P f8704464d96d4fedb16a32949e12f458: Deleting consensus metadata
I20260926 14:54:00.339933 18299 catalog_manager.cc:4841] TS a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677): tablet b09a9b1f9d044374beb7a9a1e943cc0f (table test-table [id=c459a7d1aa9649cfb8d5d47376e136c9]) successfully deleted
I20260926 14:54:00.340402 18296 catalog_manager.cc:4841] TS f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113): tablet b09a9b1f9d044374beb7a9a1e943cc0f (table test-table [id=c459a7d1aa9649cfb8d5d47376e136c9]) successfully deleted
W20260926 14:54:00.341825 18297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:44297 (error 108)
I20260926 14:54:00.341985 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18508
I20260926 14:54:00.343461 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.343340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44297 (local address 127.0.0.1:47482)
0926 14:54:00.343395 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.343433 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:54:00.343626 18297 catalog_manager.cc:4584] TS 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b09a9b1f9d044374beb7a9a1e943cc0f: Network error: Client connection negotiation failed: client connection to 127.5.15.129:44297: connect: Connection refused (error 111)
I20260926 14:54:00.347432 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18642
I20260926 14:54:00.352023 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18280
I20260926 14:54:00.357326 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:45943
--webserver_interface=127.5.15.190
--webserver_port=36489
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:00.437064 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.437268 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.437290 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.438395 18810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:00.438429 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.438443 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.438455 18810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:00.438467 18810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:00.439893 18810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:45943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:00.440073 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.440228 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.448800 18817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.448809 18816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.448948 18810 server_base.cc:1034] running on GCE node
W20260926 14:54:00.448993 18819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.449252 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.449563 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.450728 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790434440450728 us; error 35 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.453365 18810 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:54:00.453810 18810 webserver.cc:460] Webserver started at http://127.5.15.190:36489/ using document root <none> and password file <none>
I20260926 14:54:00.454042 18810 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.454123 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.455528 18810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:00.456123 18826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.456286 18810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.456388 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "07a2f401097e4b10b5596109b153d2bf"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "9ab6fc18e2bf73cc9cca2c1a9ca0f608"
server_key_iv: "7e6e6cc0b76f00eb9bf01f074882b02c"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.456755 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.464942 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.467231 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.467393 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.471216 18810 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:45943
I20260926 14:54:00.471240 18878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:45943 every 8 connection(s)
I20260926 14:54:00.471627 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:00.473167 18879 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:54:00.473806 18879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap starting.
I20260926 14:54:00.481837 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18810
I20260926 14:54:00.481980 18879 log.cc:824] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.482141 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:44297
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:45943
--webserver_port=35759
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:00.484472 18879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:00.484774 18879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap complete.
I20260926 14:54:00.487114 18879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:00.487347 18879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a2f401097e4b10b5596109b153d2bf, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.487515 18879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:00.487589 18879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:00.487643 18879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:00.487692 18879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:54:00.488648 18879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:00.488770 18879 leader_election.cc:302] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a2f401097e4b10b5596109b153d2bf; no voters:
I20260926 14:54:00.488996 18879 leader_election.cc:288] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:54:00.489029 18883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:54:00.489143 18883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 LEADER]: Becoming Leader. State: Replica: 07a2f401097e4b10b5596109b153d2bf, State: Running, Role: LEADER
I20260926 14:54:00.489279 18879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:00.489318 18883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:00.489696 18885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07a2f401097e4b10b5596109b153d2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } } }
I20260926 14:54:00.489708 18884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a2f401097e4b10b5596109b153d2bf. Latest consensus state: current_term: 2 leader_uuid: "07a2f401097e4b10b5596109b153d2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } } }
I20260926 14:54:00.489825 18885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:00.489841 18884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:00.490026 18890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:00.490909 18890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:00.491057 18890 catalog_manager.cc:1259] Loaded cluster ID: fd2cb1741d6347adaf9e65ab4c5880c5
I20260926 14:54:00.491355 18890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:00.491621 18890 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 14:54:00.496362 18890 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 14:54:00.498829 18890 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:00.499043 18890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Loaded TSK: 0
I20260926 14:54:00.499270 18890 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:54:00.564297 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.564527 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.564564 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.565653 18882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.565726 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.565774 18882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:54:00.567258 18882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:44297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=35759
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:00.567512 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.567739 18882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.573446 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.573458 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.573472 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.573613 18882 server_base.cc:1034] running on GCE node
I20260926 14:54:00.573956 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.574234 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.575385 18882 hybrid_clock.cc:648] HybridClock initialized: now 1790434440575366 us; error 37 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.583945 18882 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:54:00.584425 18882 webserver.cc:460] Webserver started at http://127.5.15.129:35759/ using document root <none> and password file <none>
I20260926 14:54:00.584661 18882 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.584749 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.586181 18882 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:00.586932 18916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.587123 18882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:00.587225 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "0c62b76e99be4851ba43d08c5b5d4d16"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "6b84ec8cfd76819a809de8bc0a10727c"
server_key_iv: "7ebfdcbddffff2b75737b443c1867872"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.587577 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.608150 18882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.610133 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.610297 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.610508 18882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:00.611027 18923 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:54:00.612424 18882 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:54:00.612474 18882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 14:54:00.612509 18882 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:54:00.613150 18882 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:54:00.613194 18882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:00.613234 18923 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap starting.
I20260926 14:54:00.620926 18882 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:44297
I20260926 14:54:00.620959 19030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:44297 every 8 connection(s)
I20260926 14:54:00.621447 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:54:00.624302 18923 log.cc:824] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.626308 18923 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:00.626605 18923 tablet_bootstrap.cc:492] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap complete.
I20260926 14:54:00.627547 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 18882
I20260926 14:54:00.627760 18923 ts_tablet_manager.cc:1397] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260926 14:54:00.627811 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:37113
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:45943
--webserver_port=33797
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:00.629383 18923 raft_consensus.cc:348] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.629684 18923 raft_consensus.cc:729] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c62b76e99be4851ba43d08c5b5d4d16, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.629856 18923 consensus_queue.cc:260] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.630134 18923 ts_tablet_manager.cc:1428] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:00.634548 19033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.622134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:38703)
0926 14:54:00.622395 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.622437 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.622964 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.624899 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.624903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.625212 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.625389 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.625394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.626640 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.626641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.627708 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.627711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.628321 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.628322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.628454 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.628998 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.629024 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.630737 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.633010 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.633016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.633018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.633230 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.633546 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.633547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.633547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.633639 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.634120 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.634124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.634228 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.634311 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.634423 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":174,"threads_started":1}
I20260926 14:54:00.634893 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.622242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:38703 (local address 127.5.15.190:45943)
0926 14:54:00.624526 (+ 2284us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.624534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.624566 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:54:00.624676 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.624681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.624720 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.624856 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.625519 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.626516 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.627792 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.628195 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.628324 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.630884 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.630888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.630889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.630900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.632910 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.633335 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.633339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.633339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.633474 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.633715 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.633716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.633716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.634047 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.634145 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.634415 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.634611 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2231,"thread_start_us":101,"threads_started":1}
I20260926 14:54:00.635550 19031 heartbeater.cc:347] Connected to a master server at 127.5.15.190:45943
I20260926 14:54:00.635624 19031 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:00.635787 19031 heartbeater.cc:511] Master 127.5.15.190:45943 requested a full tablet report, sending...
I20260926 14:54:00.636497 18843 ts_manager.cc:194] Registered new tserver with Master: 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297)
I20260926 14:54:00.637749 18843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:38703
I20260926 14:54:00.638551 19031 heartbeater.cc:503] Master 127.5.15.190:45943 was elected leader, sending a full tablet report...
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.129@KRBTEST.COM
I20260926 14:54:00.645279 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.637158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.5.15.129:44297)
0926 14:54:00.637293 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.637297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.637460 (+ 163us) server_negotiation.cc:407] Connection header received
0926 14:54:00.637570 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.637572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.637585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.637666 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.638157 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.639349 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.640076 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.640512 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.640582 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.642385 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.642391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.642391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.642397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.644106 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.644457 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.644460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.644460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.644520 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.644693 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.644694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.644694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.644850 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.644928 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.645106 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.645193 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260926 14:54:00.645284 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.637112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44297 (local address 127.0.0.1:44448)
0926 14:54:00.637357 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.637392 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.637476 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.637667 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.637672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.637913 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.637952 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.637954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.639445 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.639446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.639998 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.640000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.640587 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.640587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.640637 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.641091 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.641095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.642310 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.644186 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.644191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.644193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.644382 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.644584 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.644585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.644585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.644636 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.644918 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.644923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.644975 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.645061 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.645177 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260926 14:54:00.645676 18965 tablet_service.cc:1514] Processing DeleteTablet for tablet b09a9b1f9d044374beb7a9a1e943cc0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:00 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:44448
I20260926 14:54:00.646288 19040 tablet_replica.cc:330] stopping tablet replica
I20260926 14:54:00.646405 19040 raft_consensus.cc:2229] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:00.646479 19040 raft_consensus.cc:2258] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:00.646826 19040 ts_tablet_manager.cc:1905] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:00.648783 19040 ts_tablet_manager.cc:1918] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:00.648871 19040 log.cc:1196] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/b09a9b1f9d044374beb7a9a1e943cc0f
I20260926 14:54:00.649080 19040 ts_tablet_manager.cc:1939] T b09a9b1f9d044374beb7a9a1e943cc0f P 0c62b76e99be4851ba43d08c5b5d4d16: Deleting consensus metadata
I20260926 14:54:00.649520 18828 catalog_manager.cc:4841] TS 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297): tablet b09a9b1f9d044374beb7a9a1e943cc0f (table test-table [id=c459a7d1aa9649cfb8d5d47376e136c9]) successfully deleted
W20260926 14:54:00.712668 19036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.712855 19036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.712904 19036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.714048 19036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.714088 19036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.714112 19036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:54:00.715723 19036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:37113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=33797
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:00.715888 19036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.716051 19036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.720890 19048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.720906 19046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.721083 19036 server_base.cc:1034] running on GCE node
W20260926 14:54:00.720911 19045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.721367 19036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.721597 19036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.722784 19036 hybrid_clock.cc:648] HybridClock initialized: now 1790434440722771 us; error 24 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.725116 19036 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:54:00.725579 19036 webserver.cc:460] Webserver started at http://127.5.15.130:33797/ using document root <none> and password file <none>
I20260926 14:54:00.725797 19036 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.725875 19036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.727376 19036 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:54:00.727967 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.728160 19036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:54:00.728261 19036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "f8704464d96d4fedb16a32949e12f458"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "8d755bdf0a5b1c9d6463a235fe77f81f"
server_key_iv: "b883530b1f9f36031b754c910681b1f6"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.728574 19036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.742369 19036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.744218 19036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.744381 19036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.744604 19036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:00.744910 19036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:00.744966 19036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.745000 19036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:00.745028 19036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.750475 19036 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:37113
I20260926 14:54:00.750571 19168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:37113 every 8 connection(s)
I20260926 14:54:00.750949 19036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:00.753551 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19036
I20260926 14:54:00.753815 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:46677
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:45943
--webserver_port=43975
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:00.761022 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.751675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:47583)
0926 14:54:00.751834 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.751875 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.752344 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.752552 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.752554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.753092 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.753305 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.753310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.754260 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.754261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.755134 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.755137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.755650 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.755651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.755747 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.756213 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.756219 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.757782 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.759757 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.759763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.759764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.759979 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.760187 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.760188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.760188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.760285 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.760582 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.760586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.760690 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.760764 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.760873 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 14:54:00.761037 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.751703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:47583 (local address 127.5.15.190:45943)
0926 14:54:00.751749 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.751751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.751960 (+ 209us) server_negotiation.cc:407] Connection header received
0926 14:54:00.752425 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.752426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.752435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.752489 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.753393 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.754172 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.755220 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.755562 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.755634 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.757887 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.757889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.757889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.757899 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.759681 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.760050 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.760051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.760052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.760114 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.760351 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.760352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.760352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.760504 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.760595 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.760813 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.760909 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:54:00.761818 19169 heartbeater.cc:347] Connected to a master server at 127.5.15.190:45943
I20260926 14:54:00.761960 19169 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:00.762123 19169 heartbeater.cc:511] Master 127.5.15.190:45943 requested a full tablet report, sending...
I20260926 14:54:00.762512 18843 ts_manager.cc:194] Registered new tserver with Master: f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
I20260926 14:54:00.762980 18843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:47583
W20260926 14:54:00.834717 19172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:00.834923 19172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:00.834945 19172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:00.836076 19172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:00.836117 19172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:00.836169 19172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:54:00.837596 19172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:46677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=43975
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:00.837832 19172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:00.838078 19172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:00.842896 19180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.842900 19172 server_base.cc:1034] running on GCE node
W20260926 14:54:00.842885 19177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:00.842871 19178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:00.843279 19172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:00.843521 19172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:00.844663 19172 hybrid_clock.cc:648] HybridClock initialized: now 1790434440844653 us; error 28 us; skew 500 ppm
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:00.847097 19172 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:54:00.847532 19172 webserver.cc:460] Webserver started at http://127.5.15.131:43975/ using document root <none> and password file <none>
I20260926 14:54:00.847721 19172 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:00.847770 19172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:00.849073 19172 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:00.849877 19187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.850061 19172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:00.850157 19172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "a405277676a34f93ad239d2059f7b9e0"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-s4p1"
server_key: "f17383f27752fd930ebc06be63cd9a3a"
server_key_iv: "05e11d947b03d15ba00e59e8ba7f5bb8"
server_key_version: "encryptionkey@0"
I20260926 14:54:00.850489 19172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:00.886813 19172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:00.888705 19172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:00.888877 19172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:00.889158 19172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:00.889513 19172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:00.889567 19172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.889611 19172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:00.889637 19172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.895591 19172 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46677
I20260926 14:54:00.895640 19300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46677 every 8 connection(s)
I20260926 14:54:00.895954 19172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:54:00.899319 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19172
Sep 26 14:54:00 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434440, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:00.905429 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.896661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:33745 (local address 127.5.15.190:45943)
0926 14:54:00.896729 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.896731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.896946 (+ 215us) server_negotiation.cc:407] Connection header received
0926 14:54:00.897402 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.897403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.897414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.897463 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.898381 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.898932 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.900033 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.900355 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.900424 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.902702 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.902704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.902704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.902711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.904041 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.904436 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.904438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.904438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.904497 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.904770 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.904771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.904772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.904883 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.904968 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.905265 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.905340 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:54:00.905618 19303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.896670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:33745)
0926 14:54:00.896825 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.896861 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.897320 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.897535 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.897538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.898101 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:00.898305 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.898310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.899002 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.899003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.899976 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.899979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.900441 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.900442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.900544 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.901042 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:00.901049 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:00.902617 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:00.904120 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.904127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.904129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.904348 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.904588 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:00.904589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:00.904590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:00.904701 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:00.904963 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:00.904970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:00.905128 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.905232 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.905383 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260926 14:54:00.905997 19301 heartbeater.cc:347] Connected to a master server at 127.5.15.190:45943
I20260926 14:54:00.906119 19301 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:00.906306 19301 heartbeater.cc:511] Master 127.5.15.190:45943 requested a full tablet report, sending...
I20260926 14:54:00.906551 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.899667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.5.15.190:45943)
0926 14:54:00.899783 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.899785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.899810 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:54:00.899907 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.899908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.899919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.899976 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:00.900644 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.901110 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.901998 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.902219 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.902311 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.903778 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:00.903780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:00.903780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:00.903787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:00.905225 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.905603 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.905604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.905604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.905684 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:00.906012 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:00.906013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:00.906013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:00.906156 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:00.906298 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.906378 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.906436 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260926 14:54:00.906840 18843 ts_manager.cc:194] Registered new tserver with Master: a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677)
I20260926 14:54:00.907303 18843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:33745
I20260926 14:54:00.907315 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:54:00.909878 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.907629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.5.15.190:45943)
0926 14:54:00.907664 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.907666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.907813 (+ 147us) server_negotiation.cc:407] Connection header received
0926 14:54:00.907911 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.907912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.907937 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.907991 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:00.908123 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.908539 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.909232 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.909412 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.909458 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.909580 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:00.909679 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:00.909745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.909774 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.909789 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:54:00.910108 18843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:54:00.911643 18843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:54:00.913561 18965 tablet_service.cc:1467] Processing CreateTablet for tablet dcc7ffcce2d94b049d3a1f883d4111b8 (DEFAULT_TABLE table=test-table [id=f6a2287839bd4d2d969ada55bcb6c90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:00.913758 18965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc7ffcce2d94b049d3a1f883d4111b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.915263 18923 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap starting.
I20260926 14:54:00.915525 18923 tablet_bootstrap.cc:654] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.916227 18923 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: No bootstrap required, opened a new log
I20260926 14:54:00.916272 18923 ts_tablet_manager.cc:1397] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:54:00.916420 18923 raft_consensus.cc:348] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.916488 18923 raft_consensus.cc:374] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.916523 18923 raft_consensus.cc:729] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c62b76e99be4851ba43d08c5b5d4d16, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.916591 18923 consensus_queue.cc:260] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.916702 18923 ts_tablet_manager.cc:1428] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:00.917208 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.913270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46677 (local address 127.0.0.1:36360)
0926 14:54:00.913433 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.913444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.913486 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.913890 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.913891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.913893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.913940 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.913941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.914867 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.914868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.916025 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.916028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.916980 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.916981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.917060 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.917079 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.917108 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.917144 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":2}
I20260926 14:54:00.917677 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.913562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.5.15.131:46677)
0926 14:54:00.913736 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.913740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.913750 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:54:00.913786 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.913787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.913799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.913885 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.913991 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.914788 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.916116 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.916867 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.917466 (+ 599us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.917530 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.917555 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.917598 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260926 14:54:00.918257 19235 tablet_service.cc:1467] Processing CreateTablet for tablet dcc7ffcce2d94b049d3a1f883d4111b8 (DEFAULT_TABLE table=test-table [id=f6a2287839bd4d2d969ada55bcb6c90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:00.918498 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.913301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:37113 (local address 127.0.0.1:55412)
0926 14:54:00.914839 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:00.914850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:00.914908 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:00.915081 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:00.915082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:00.915084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:00.915129 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.915130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.916265 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.916265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.917514 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:00.917517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.918259 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.918261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:00.918322 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.918327 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:54:00.918397 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:54:00.918436 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"mutex_wait_us":12,"thread_start_us":91,"threads_started":1}
I20260926 14:54:00.918547 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:00.913396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55412 (local address 127.5.15.130:37113)
0926 14:54:00.913578 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:00.913583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:00.914905 (+ 1322us) server_negotiation.cc:407] Connection header received
0926 14:54:00.914973 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:00.914975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:00.914990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:00.915070 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:00.915189 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.916158 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.917592 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:00.918179 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:00.918235 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:00.918267 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:00.918403 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 14:54:00.918466 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260926 14:54:00.918587 19235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc7ffcce2d94b049d3a1f883d4111b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.919307 19103 tablet_service.cc:1467] Processing CreateTablet for tablet dcc7ffcce2d94b049d3a1f883d4111b8 (DEFAULT_TABLE table=test-table [id=f6a2287839bd4d2d969ada55bcb6c90f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:00.919593 19103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc7ffcce2d94b049d3a1f883d4111b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:00.921363 19311 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Bootstrap starting.
I20260926 14:54:00.922056 19311 tablet_bootstrap.cc:654] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.922271 19312 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Bootstrap starting.
I20260926 14:54:00.922636 19311 log.cc:824] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.922959 19312 tablet_bootstrap.cc:654] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:00.923466 19311 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: No bootstrap required, opened a new log
I20260926 14:54:00.923514 19312 log.cc:824] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:00.923542 19311 ts_tablet_manager.cc:1397] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:00.924201 19312 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: No bootstrap required, opened a new log
I20260926 14:54:00.924286 19312 ts_tablet_manager.cc:1397] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:00.925537 19311 raft_consensus.cc:348] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.925716 19311 raft_consensus.cc:374] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.925752 19311 raft_consensus.cc:729] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a405277676a34f93ad239d2059f7b9e0, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.925918 19311 consensus_queue.cc:260] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.926213 19311 ts_tablet_manager.cc:1428] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:54:00.926226 19301 heartbeater.cc:503] Master 127.5.15.190:45943 was elected leader, sending a full tablet report...
I20260926 14:54:00.926296 19312 raft_consensus.cc:348] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.926438 19312 raft_consensus.cc:374] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:00.926461 19312 raft_consensus.cc:729] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8704464d96d4fedb16a32949e12f458, State: Initialized, Role: FOLLOWER
I20260926 14:54:00.926568 19312 consensus_queue.cc:260] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:00.926852 19169 heartbeater.cc:503] Master 127.5.15.190:45943 was elected leader, sending a full tablet report...
I20260926 14:54:00.926859 19312 ts_tablet_manager.cc:1428] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 14:54:01.001583 19170 tablet.cc:2369] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:54:01.122246 19032 tablet.cc:2369] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:01.134261 19037 raft_consensus.cc:482] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:01.134415 19037 raft_consensus.cc:504] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:01.134845 19037 leader_election.cc:288] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
I20260926 14:54:01.138911 19308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.135212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:60857 (local address 127.5.15.130:37113)
0926 14:54:01.135253 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.135255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.135271 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:54:01.135341 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.135342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.135354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.135421 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.135622 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.136362 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.137925 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.138474 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.138536 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.138544 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.138786 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.138830 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:54:01.138939 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.135147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:40423 (local address 127.5.15.131:46677)
0926 14:54:01.135193 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.135195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.135227 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:54:01.135342 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.135343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.135354 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.135421 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:01.135622 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.136366 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.137993 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.138554 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.138636 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.138645 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.138824 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.138870 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:54:01.138958 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.135051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:37113 (local address 127.5.15.129:60857)
0926 14:54:01.135174 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.135182 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.135256 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.135439 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.135440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.135444 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.135544 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.135546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.136449 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.136449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.137846 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.137850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.138553 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.138554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.138643 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.138673 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.138731 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.138832 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":75,"thread_start_us":67,"threads_started":1}
I20260926 14:54:01.138968 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.135036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46677 (local address 127.5.15.129:40423)
0926 14:54:01.135155 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.135168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.135256 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.135439 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:01.135440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:01.135443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:01.135544 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.135546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.136448 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.136449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.137847 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:01.137850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.138636 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.138637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:01.138700 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.138703 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:54:01.138812 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:54:01.138855 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":113,"thread_start_us":74,"threads_started":1}
I20260926 14:54:01.139391 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc7ffcce2d94b049d3a1f883d4111b8" candidate_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8704464d96d4fedb16a32949e12f458" is_pre_election: true
I20260926 14:54:01.139449 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc7ffcce2d94b049d3a1f883d4111b8" candidate_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405277676a34f93ad239d2059f7b9e0" is_pre_election: true
I20260926 14:54:01.139550 19123 raft_consensus.cc:2454] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c62b76e99be4851ba43d08c5b5d4d16 in term 0.
I20260926 14:54:01.139611 19255 raft_consensus.cc:2454] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c62b76e99be4851ba43d08c5b5d4d16 in term 0.
I20260926 14:54:01.139798 18917 leader_election.cc:302] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16, f8704464d96d4fedb16a32949e12f458; no voters:
I20260926 14:54:01.139917 19037 raft_consensus.cc:2781] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:54:01.139974 19037 raft_consensus.cc:482] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:54:01.139995 19037 raft_consensus.cc:3037] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.140688 19037 raft_consensus.cc:504] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:01.140822 19037 leader_election.cc:288] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 1 election: Requested vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
I20260926 14:54:01.140959 19255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc7ffcce2d94b049d3a1f883d4111b8" candidate_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a405277676a34f93ad239d2059f7b9e0"
I20260926 14:54:01.140981 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc7ffcce2d94b049d3a1f883d4111b8" candidate_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8704464d96d4fedb16a32949e12f458"
I20260926 14:54:01.141047 19255 raft_consensus.cc:3037] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.141047 19123 raft_consensus.cc:3037] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:01.141937 19255 raft_consensus.cc:2454] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c62b76e99be4851ba43d08c5b5d4d16 in term 1.
I20260926 14:54:01.141937 19123 raft_consensus.cc:2454] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c62b76e99be4851ba43d08c5b5d4d16 in term 1.
I20260926 14:54:01.142119 18917 leader_election.cc:302] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16, f8704464d96d4fedb16a32949e12f458; no voters:
I20260926 14:54:01.142239 19037 raft_consensus.cc:2781] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:01.142398 19037 raft_consensus.cc:686] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 LEADER]: Becoming Leader. State: Replica: 0c62b76e99be4851ba43d08c5b5d4d16, State: Running, Role: LEADER
I20260926 14:54:01.142454 19037 consensus_queue.cc:237] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:01.143277 18842 catalog_manager.cc:5495] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } health_report { overall_health: HEALTHY } } }
W20260926 14:54:01.146440 19302 tablet.cc:2369] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:01.198673 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.196382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.5.15.129:44297)
0926 14:54:01.196487 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:01.196489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:01.196502 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:54:01.196538 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:01.196538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:01.196558 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:01.196627 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:01.196740 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.197201 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.197877 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:01.198101 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:01.198151 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:01.198334 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:01.198434 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:01.198488 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:01.198537 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 14:54:01.198555 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260926 14:54:01.200244 19255 raft_consensus.cc:1261] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Refusing update from remote peer 0c62b76e99be4851ba43d08c5b5d4d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:01.200250 19123 raft_consensus.cc:1261] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Refusing update from remote peer 0c62b76e99be4851ba43d08c5b5d4d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:01.200614 19037 consensus_queue.cc:1035] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:01.200747 19319 consensus_queue.cc:1035] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:01.203518 19324 mvcc.cc:205] Tried to move back new op lower bound from 7333619471153623040 to 7333619470920151040. Current Snapshot: MvccSnapshot[applied={T|T < 7333619471153623040}]
I20260926 14:54:01.205355 18843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:01.205435 18843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:01.206111 18843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Sending DeleteTablet for 3 replicas of tablet dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:01.206413 18965 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc7ffcce2d94b049d3a1f883d4111b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:01 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:44448
I20260926 14:54:01.206502 19235 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc7ffcce2d94b049d3a1f883d4111b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:01 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:36360
I20260926 14:54:01.206550 19103 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc7ffcce2d94b049d3a1f883d4111b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:01 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:55412
I20260926 14:54:01.206784 19334 tablet_replica.cc:330] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: stopping tablet replica
I20260926 14:54:01.206805 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18882
I20260926 14:54:01.206912 19334 raft_consensus.cc:2229] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:01.207072 19334 raft_consensus.cc:2258] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:01.208657 19334 ts_tablet_manager.cc:1905] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:01.210877 19333 tablet_replica.cc:330] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: stopping tablet replica
I20260926 14:54:01.210893 19334 ts_tablet_manager.cc:1918] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:01.210953 19334 log.cc:1196] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:01.211020 19333 raft_consensus.cc:2229] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:01.211143 19333 raft_consensus.cc:2258] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:01.211251 19334 ts_tablet_manager.cc:1939] T dcc7ffcce2d94b049d3a1f883d4111b8 P f8704464d96d4fedb16a32949e12f458: Deleting consensus metadata
I20260926 14:54:01.211589 19333 ts_tablet_manager.cc:1905] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:01.211694 18827 catalog_manager.cc:4841] TS f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113): tablet dcc7ffcce2d94b049d3a1f883d4111b8 (table test-table [id=f6a2287839bd4d2d969ada55bcb6c90f]) successfully deleted
W20260926 14:54:01.212792 18828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:44297 (error 108)
I20260926 14:54:01.212987 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19036
I20260926 14:54:01.213333 19333 ts_tablet_manager.cc:1918] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:01.213397 19333 log.cc:1196] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:01.213656 19333 ts_tablet_manager.cc:1939] T dcc7ffcce2d94b049d3a1f883d4111b8 P a405277676a34f93ad239d2059f7b9e0: Deleting consensus metadata
I20260926 14:54:01.213865 19307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.213163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:44297 (local address 127.0.0.1:44466)
0926 14:54:01.213803 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.213839 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622}
I20260926 14:54:01.214032 18830 catalog_manager.cc:4841] TS a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677): tablet dcc7ffcce2d94b049d3a1f883d4111b8 (table test-table [id=f6a2287839bd4d2d969ada55bcb6c90f]) successfully deleted
W20260926 14:54:01.213986 18828 catalog_manager.cc:4584] TS 0c62b76e99be4851ba43d08c5b5d4d16 (127.5.15.129:44297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcc7ffcce2d94b049d3a1f883d4111b8: Network error: Client connection negotiation failed: client connection to 127.5.15.129:44297: connect: Connection refused (error 111)
I20260926 14:54:01.218423 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19172
I20260926 14:54:01.223601 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 18810
I20260926 14:54:01.228530 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:45943
--webserver_interface=127.5.15.190
--webserver_port=36489
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:01.308372 19336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.308569 19336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.308590 19336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.309747 19336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:01.309788 19336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.309803 19336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.309854 19336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:01.309870 19336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:01.311352 19336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:45943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=36489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:01.311622 19336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.311863 19336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.316638 19342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.316699 19341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.316699 19344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.317102 19336 server_base.cc:1034] running on GCE node
I20260926 14:54:01.317335 19336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.317584 19336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.318723 19336 hybrid_clock.cc:648] HybridClock initialized: now 1790434441318704 us; error 36 us; skew 500 ppm
Sep 26 14:54:01 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:01.321097 19336 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:54:01.321560 19336 webserver.cc:460] Webserver started at http://127.5.15.190:36489/ using document root <none> and password file <none>
I20260926 14:54:01.321794 19336 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.321861 19336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.323220 19336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.323884 19351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.324132 19336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.324230 19336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "07a2f401097e4b10b5596109b153d2bf"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "9ab6fc18e2bf73cc9cca2c1a9ca0f608"
server_key_iv: "7e6e6cc0b76f00eb9bf01f074882b02c"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.324546 19336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.339026 19336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.340955 19336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.341150 19336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.345021 19336 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:45943
I20260926 14:54:01.345096 19403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:45943 every 8 connection(s)
I20260926 14:54:01.345438 19336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:01.347121 19404 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:54:01.347795 19404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap starting.
I20260926 14:54:01.353346 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19336
I20260926 14:54:01.353618 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:44297
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:45943
--webserver_port=35759
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:01.355973 19404 log.cc:824] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:01.358124 19404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:01.358412 19404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf: Bootstrap complete.
I20260926 14:54:01.360642 19404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:01.360882 19404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a2f401097e4b10b5596109b153d2bf, State: Initialized, Role: FOLLOWER
I20260926 14:54:01.361093 19404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:01.361191 19404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:01.361245 19404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:01.361292 19404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:54:01.362228 19404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:01.362345 19404 leader_election.cc:302] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a2f401097e4b10b5596109b153d2bf; no voters:
I20260926 14:54:01.362524 19404 leader_election.cc:288] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:54:01.362572 19408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:54:01.362721 19408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [term 3 LEADER]: Becoming Leader. State: Replica: 07a2f401097e4b10b5596109b153d2bf, State: Running, Role: LEADER
I20260926 14:54:01.362784 19404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:01.362882 19408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } }
I20260926 14:54:01.363224 19408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a2f401097e4b10b5596109b153d2bf. Latest consensus state: current_term: 3 leader_uuid: "07a2f401097e4b10b5596109b153d2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } } }
I20260926 14:54:01.363361 19408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:01.363225 19409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "07a2f401097e4b10b5596109b153d2bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a2f401097e4b10b5596109b153d2bf" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 45943 } } }
I20260926 14:54:01.363483 19409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a2f401097e4b10b5596109b153d2bf [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:01.363566 19417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:01.364319 19417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:01.364408 19417 catalog_manager.cc:1259] Loaded cluster ID: fd2cb1741d6347adaf9e65ab4c5880c5
I20260926 14:54:01.364630 19417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 14:54:01.364846 19417 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdbc6cf79ad google::LogMessage::Fail() at ??:0
@ 0x7fdbc6cfa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdbc6cf74de google::LogMessage::Flush() at ??:0
@ 0x7fdbc6cf8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdbcec180f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdbcec218b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdbc8bc50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdbc8bba954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdbc9a1e6db start_thread at ??:0
@ 0x7fdbc57f561f clone at ??:0
W20260926 14:54:01.447438 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.447661 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.447698 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.448916 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.448982 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.449020 19406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:54:01.450423 19406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:44297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=35759
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:01.450690 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.450941 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.455792 19441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.455804 19443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.456161 19406 server_base.cc:1034] running on GCE node
W20260926 14:54:01.455829 19440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.456387 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.456651 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.457832 19406 hybrid_clock.cc:648] HybridClock initialized: now 1790434441457819 us; error 27 us; skew 500 ppm
Sep 26 14:54:01 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:01.460609 19406 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:54:01.461139 19406 webserver.cc:460] Webserver started at http://127.5.15.129:35759/ using document root <none> and password file <none>
I20260926 14:54:01.461376 19406 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.461443 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.462852 19406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:54:01.463554 19450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.463836 19406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.463936 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "0c62b76e99be4851ba43d08c5b5d4d16"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "6b84ec8cfd76819a809de8bc0a10727c"
server_key_iv: "7ebfdcbddffff2b75737b443c1867872"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.464429 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.482712 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.485556 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.485731 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.486037 19406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:01.486626 19457 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:54:01.488121 19406 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:54:01.488221 19406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 14:54:01.488276 19406 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:54:01.489657 19406 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:54:01.489732 19406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:01.489845 19457 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap starting.
I20260926 14:54:01.497627 19406 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:44297
I20260926 14:54:01.497821 19564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:44297 every 8 connection(s)
I20260926 14:54:01.498139 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:54:01.498804 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19406
I20260926 14:54:01.499119 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:37113
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:45943
--webserver_port=33797
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:01.503454 19457 log.cc:824] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:01.505779 19457 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:01.506108 19457 tablet_bootstrap.cc:492] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Bootstrap complete.
I20260926 14:54:01.507186 19457 ts_tablet_manager.cc:1397] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260926 14:54:01.508525 19457 raft_consensus.cc:348] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:01.508771 19457 raft_consensus.cc:729] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c62b76e99be4851ba43d08c5b5d4d16, State: Initialized, Role: FOLLOWER
I20260926 14:54:01.508962 19457 consensus_queue.cc:260] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:01.509274 19457 ts_tablet_manager.cc:1428] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260926 14:54:01.598994 19568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.599205 19568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.599243 19568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.600392 19568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.600459 19568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.600503 19568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:54:01.601902 19568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:37113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=33797
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:01.602154 19568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.602384 19568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.607216 19575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.607388 19568 server_base.cc:1034] running on GCE node
W20260926 14:54:01.607228 19578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.607245 19576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.607654 19568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.607935 19568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.609102 19568 hybrid_clock.cc:648] HybridClock initialized: now 1790434441609068 us; error 50 us; skew 500 ppm
Sep 26 14:54:01 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:01.611681 19568 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:54:01.612111 19568 webserver.cc:460] Webserver started at http://127.5.15.130:33797/ using document root <none> and password file <none>
I20260926 14:54:01.612344 19568 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.612389 19568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.613799 19568 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.614445 19585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.614630 19568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.614714 19568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "f8704464d96d4fedb16a32949e12f458"
format_stamp: "Formatted at 2026-09-26 14:53:59 on dist-test-slave-s4p1"
server_key: "8d755bdf0a5b1c9d6463a235fe77f81f"
server_key_iv: "b883530b1f9f36031b754c910681b1f6"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.615079 19568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260926 14:54:01.626140 19567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:01.498959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:59343)
0926 14:54:01.499217 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.499279 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:01.500066 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:01.625962 (+125896us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
W20260926 14:54:01.626376 19565 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:45943 (0 consecutive failures): Network error: Failed to ping master at 127.5.15.190:45943: Client connection negotiation failed: client connection to 127.5.15.190:45943: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:54:01.626678 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.626563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:35819)
0926 14:54:01.626614 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.626661 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:54:01.627007 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.626906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:60463)
0926 14:54:01.626950 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.626972 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 14:54:01.627116 19565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:54:01.634555 19568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.636458 19568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.636638 19568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.636853 19568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:01.637197 19568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:01.637248 19568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.637290 19568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:01.637315 19568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.643127 19568 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:37113
I20260926 14:54:01.643198 19698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:37113 every 8 connection(s)
I20260926 14:54:01.643529 19568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:01.644531 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.644207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:42607)
0926 14:54:01.644340 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.644400 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
W20260926 14:54:01.644939 19699 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:45943 (0 consecutive failures): Network error: Failed to ping master at 127.5.15.190:45943: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
I20260926 14:54:01.645192 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.645106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:43725)
0926 14:54:01.645152 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.645176 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:54:01.645457 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.645386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:40749)
0926 14:54:01.645424 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.645443 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:54:01.645586 19699 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:54:01.647315 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19568
I20260926 14:54:01.647595 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:46677
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:45943
--webserver_port=43975
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:01.727943 19702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:01.728144 19702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:01.728168 19702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:01.729337 19702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:01.729379 19702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:01.729432 19702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:54:01.730898 19702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:46677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=43975
--tserver_master_addrs=127.5.15.190:45943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:01.731146 19702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:01.731384 19702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:01.736248 19710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.736279 19707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:01.736361 19708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:01.736434 19702 server_base.cc:1034] running on GCE node
I20260926 14:54:01.736608 19702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:01.736797 19702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:01.737933 19702 hybrid_clock.cc:648] HybridClock initialized: now 1790434441737924 us; error 28 us; skew 500 ppm
Sep 26 14:54:01 dist-test-slave-s4p1 krb5kdc[18267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:01.740489 19702 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:54:01.740895 19702 webserver.cc:460] Webserver started at http://127.5.15.131:43975/ using document root <none> and password file <none>
I20260926 14:54:01.741107 19702 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:01.741153 19702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:01.742439 19702 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.743089 19717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.743261 19702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:01.743359 19702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "a405277676a34f93ad239d2059f7b9e0"
format_stamp: "Formatted at 2026-09-26 14:54:00 on dist-test-slave-s4p1"
server_key: "f17383f27752fd930ebc06be63cd9a3a"
server_key_iv: "05e11d947b03d15ba00e59e8ba7f5bb8"
server_key_version: "encryptionkey@0"
I20260926 14:54:01.743746 19702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:01.767376 19702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:01.769860 19702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:01.770022 19702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:01.770259 19702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:01.770562 19702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:01.770613 19702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.770655 19702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:01.770684 19702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:01.776384 19702 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46677
I20260926 14:54:01.776434 19830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46677 every 8 connection(s)
I20260926 14:54:01.776759 19702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:54:01.777753 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.777377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:36223)
0926 14:54:01.777551 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.777617 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
W20260926 14:54:01.778198 19831 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:45943 (0 consecutive failures): Network error: Failed to ping master at 127.5.15.190:45943: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
I20260926 14:54:01.778491 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.778393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:59773)
0926 14:54:01.778447 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.778473 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:54:01.778793 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:01.778703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:34151)
0926 14:54:01.778757 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:01.778779 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 14:54:01.778905 19831 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:54:01.782390 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19702
I20260926 14:54:02.627640 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.627417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:52731)
0926 14:54:02.627537 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.627575 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260926 14:54:02.646166 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.645884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:45031)
0926 14:54:02.646079 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.646114 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":53,"threads_started":1}
I20260926 14:54:02.779397 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.779219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:58185)
0926 14:54:02.779330 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.779357 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260926 14:54:02.933558 19838 raft_consensus.cc:482] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:02.933713 19838 raft_consensus.cc:504] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:02.934063 19838 leader_election.cc:288] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
Sep 26 14:54:02 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.130@KRBTEST.COM
Sep 26 14:54:02 dist-test-slave-s4p1 krb5kdc[18267](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434441, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.131@KRBTEST.COM
I20260926 14:54:02.944491 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.934352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:44999 (local address 127.5.15.131:46677)
0926 14:54:02.934514 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.934526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.934572 (+ 46us) server_negotiation.cc:407] Connection header received
0926 14:54:02.935080 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.935082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.935126 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.935237 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:02.936027 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.936853 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.937831 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.938249 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.938371 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.941069 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:02.941073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:02.941074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:02.941085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:02.942861 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:02.943237 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:02.943240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:02.943241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:02.943352 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:02.943583 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:02.943584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:02.943584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:02.943911 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:02.944001 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.944261 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.944383 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 14:54:02.944478 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.934245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:37113 (local address 127.5.15.129:42805)
0926 14:54:02.934379 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.934391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.934462 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.935334 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.935334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.935693 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:02.935884 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.935889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.936973 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.936974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.937736 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.937738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.938312 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.938313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.938421 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.939063 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:02.939072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:02.940588 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:02.942665 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:02.942672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:02.942674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:02.943017 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:02.943270 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:02.943271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:02.943272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:02.943391 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:02.943976 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:02.943983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:02.944102 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.944202 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.944381 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":61,"thread_start_us":61,"threads_started":1}
I20260926 14:54:02.944527 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.934379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:42805 (local address 127.5.15.130:37113)
0926 14:54:02.934561 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:02.934570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:02.934596 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:54:02.935080 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:02.935083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:02.935126 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:02.935286 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:02.935995 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.936853 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.937823 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.938227 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.938331 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.940693 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:02.940697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:02.940697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:02.940712 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:02.942586 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:02.943092 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:02.943095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:02.943095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:02.943205 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:02.943550 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:02.943551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:02.943552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:02.943905 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:02.943991 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:02.944269 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 14:54:02.944416 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260926 14:54:02.944597 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:02.934232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46677 (local address 127.5.15.129:44999)
0926 14:54:02.934315 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:02.934325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:02.934380 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:02.935309 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:02.935312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:02.935695 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:02.935884 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.935890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.936972 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.936973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.937737 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:02.937739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:02.938352 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:02.938353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:02.938429 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:02.939139 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:02.939143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:02.940961 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:02.942945 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:02.942946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:02.942946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:02.943152 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:02.943439 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:02.943440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:02.943440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:02.943507 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:02.943998 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:02.943999 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:02.944103 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 14:54:02.944210 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:54:02.944458 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":133}
W20260926 14:54:02.945300 19451 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
W20260926 14:54:02.945835 19454 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:02.945928 19454 leader_election.cc:302] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16; no voters: a405277676a34f93ad239d2059f7b9e0, f8704464d96d4fedb16a32949e12f458
I20260926 14:54:02.946013 19838 raft_consensus.cc:2726] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:54:03.628319 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:03.628108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:37491)
0926 14:54:03.628239 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:03.628273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260926 14:54:03.646899 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:03.646638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:41309)
0926 14:54:03.646816 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:03.646863 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":55,"threads_started":1}
I20260926 14:54:03.779955 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:03.779763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:44527)
0926 14:54:03.779887 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:03.779918 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260926 14:54:04.556354 19845 raft_consensus.cc:482] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:04.556473 19845 raft_consensus.cc:504] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:04.556684 19845 leader_election.cc:288] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
W20260926 14:54:04.557052 19454 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
W20260926 14:54:04.557133 19451 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:04.557236 19451 leader_election.cc:302] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16; no voters: a405277676a34f93ad239d2059f7b9e0, f8704464d96d4fedb16a32949e12f458
I20260926 14:54:04.557315 19845 raft_consensus.cc:2726] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:54:04.628957 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:04.628756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:57965)
0926 14:54:04.628877 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:04.628910 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260926 14:54:04.647526 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:04.647351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:54655)
0926 14:54:04.647459 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:04.647491 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260926 14:54:04.780610 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:04.780416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:53947)
0926 14:54:04.780546 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:04.780577 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260926 14:54:05.629531 19849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:05.629336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:53251)
0926 14:54:05.629454 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:05.629485 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260926 14:54:05.648052 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:05.647900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:60283)
0926 14:54:05.647994 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:05.648024 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260926 14:54:05.781245 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:05.781037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:56749)
0926 14:54:05.781163 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:05.781196 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260926 14:54:06.630235 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:06.630010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:52195)
0926 14:54:06.630139 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:06.630172 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260926 14:54:06.648627 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:06.648460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:48793)
0926 14:54:06.648565 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:06.648594 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260926 14:54:06.781824 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:06.781638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:32879)
0926 14:54:06.781760 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:06.781790 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260926 14:54:07.122016 19855 raft_consensus.cc:482] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:07.122134 19855 raft_consensus.cc:504] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:07.122328 19855 leader_election.cc:288] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
W20260926 14:54:07.122668 19454 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
W20260926 14:54:07.122803 19451 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:07.122871 19451 leader_election.cc:302] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16; no voters: a405277676a34f93ad239d2059f7b9e0, f8704464d96d4fedb16a32949e12f458
I20260926 14:54:07.122941 19855 raft_consensus.cc:2726] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:54:07.630858 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:07.630666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:57231)
0926 14:54:07.630786 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:07.630815 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":54,"threads_started":1}
I20260926 14:54:07.649274 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:07.649101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:40615)
0926 14:54:07.649211 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:07.649241 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260926 14:54:07.782470 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:07.782269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:51433)
0926 14:54:07.782402 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:07.782436 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260926 14:54:08.631465 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:08.631266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:40243)
0926 14:54:08.631384 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:08.631418 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260926 14:54:08.649865 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:08.649667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:54223)
0926 14:54:08.649799 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:08.649830 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260926 14:54:08.783180 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:08.782975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:37595)
0926 14:54:08.783111 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:08.783143 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260926 14:54:09.632114 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:09.631897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:47779)
0926 14:54:09.632013 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:09.632046 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260926 14:54:09.650570 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:09.650368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:38545)
0926 14:54:09.650503 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:09.650533 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260926 14:54:09.783802 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:09.783596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:60827)
0926 14:54:09.783724 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:09.783756 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260926 14:54:10.257989 19865 raft_consensus.cc:482] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:10.258101 19865 raft_consensus.cc:504] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a405277676a34f93ad239d2059f7b9e0" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46677 } } peers { permanent_uuid: "f8704464d96d4fedb16a32949e12f458" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 37113 } } peers { permanent_uuid: "0c62b76e99be4851ba43d08c5b5d4d16" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 44297 } }
I20260926 14:54:10.258328 19865 leader_election.cc:288] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677), f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113)
W20260926 14:54:10.258709 19454 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a405277676a34f93ad239d2059f7b9e0 (127.5.15.131:46677): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
W20260926 14:54:10.258821 19451 leader_election.cc:341] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8704464d96d4fedb16a32949e12f458 (127.5.15.130:37113): Not found: Tablet not found: dcc7ffcce2d94b049d3a1f883d4111b8
I20260926 14:54:10.258893 19451 leader_election.cc:302] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c62b76e99be4851ba43d08c5b5d4d16; no voters: a405277676a34f93ad239d2059f7b9e0, f8704464d96d4fedb16a32949e12f458
I20260926 14:54:10.258967 19865 raft_consensus.cc:2726] T dcc7ffcce2d94b049d3a1f883d4111b8 P 0c62b76e99be4851ba43d08c5b5d4d16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:54:10.632823 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:10.632519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:33041)
0926 14:54:10.632742 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:10.632776 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":59,"threads_started":1}
I20260926 14:54:10.651149 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:10.650922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:53197)
0926 14:54:10.651075 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:10.651101 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260926 14:54:10.784400 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:10.784214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:37919)
0926 14:54:10.784334 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:10.784366 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
I20260926 14:54:11.633635 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:11.633448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:40623)
0926 14:54:11.633562 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:11.633591 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260926 14:54:11.651778 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:11.651544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:38059)
0926 14:54:11.651689 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:11.651744 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260926 14:54:11.785041 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:11.784840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:40269)
0926 14:54:11.784970 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:11.785005 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260926 14:54:12.634234 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:12.634029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:32807)
0926 14:54:12.634153 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:12.634190 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260926 14:54:12.652433 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:12.652226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:42029)
0926 14:54:12.652354 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:12.652391 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260926 14:54:12.785810 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:12.785602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:41877)
0926 14:54:12.785735 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:12.785770 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260926 14:54:13.634987 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:13.634758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:35327)
0926 14:54:13.634885 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:13.634941 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260926 14:54:13.653196 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:13.652986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:47419)
0926 14:54:13.653119 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:13.653153 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260926 14:54:13.786435 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:13.786222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:59293)
0926 14:54:13.786364 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:13.786397 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260926 14:54:14.635594 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:14.635379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:33525)
0926 14:54:14.635513 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:14.635548 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260926 14:54:14.653890 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:14.653716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:57075)
0926 14:54:14.653825 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:14.653855 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260926 14:54:14.787135 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:14.786956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:51025)
0926 14:54:14.787070 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:14.787100 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260926 14:54:15.636195 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:15.635991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:50609)
0926 14:54:15.636116 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:15.636149 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260926 14:54:15.654469 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:15.654304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:57551)
0926 14:54:15.654407 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:15.654436 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260926 14:54:15.787663 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:15.787506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:44789)
0926 14:54:15.787606 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:15.787632 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20260926 14:54:16.636862 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:16.636674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.129:57093)
0926 14:54:16.636786 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:16.636817 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 14:54:16.655107 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:16.654928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.130:45627)
0926 14:54:16.655046 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:16.655076 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260926 14:54:16.784119 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19406
I20260926 14:54:16.788277 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:16.788107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:45943 (local address 127.5.15.131:60317)
0926 14:54:16.788218 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:16.788244 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:45943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260926 14:54:16.788820 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19568
I20260926 14:54:16.793128 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19702
2026-09-26T14:54:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18401 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:54:16 dist-test-slave-s4p1 krb5kdc[19889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:54:16 dist-test-slave-s4p1 krb5kdc[19889](info): set up 2 sockets
Sep 26 14:54:16 dist-test-slave-s4p1 krb5kdc[19889](info): commencing operation
krb5kdc: starting...
W20260926 14:54:17.807046 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:54:17 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:54:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:54:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:54:17.917791 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37971
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:17.996484 19902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:17.996706 19902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:17.996739 19902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:17.998102 19902 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:54:17.998180 19902 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:54:17.998205 19902 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:54:17.998224 19902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:17.998257 19902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:17.998283 19902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:17.998310 19902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:17.998337 19902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:17.999930 19902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:37971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:18.000198 19902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:18.000397 19902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:18.004938 19910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.004940 19908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.005129 19902 server_base.cc:1034] running on GCE node
W20260926 14:54:18.004992 19907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.005450 19902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:18.005751 19902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:18.006906 19902 hybrid_clock.cc:648] HybridClock initialized: now 1790434458006902 us; error 35 us; skew 500 ppm
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:18.009766 19902 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:54:18.010309 19902 webserver.cc:460] Webserver started at http://127.5.15.190:37471/ using document root <none> and password file <none>
I20260926 14:54:18.010576 19902 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:18.010659 19902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:18.010835 19902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:18.011970 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "b45a4d9f2df04f908d7ff4ef7e2df789"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "08d8ada65ac91d5a1bfe4abf9fd8fbee"
server_key_iv: "ffd6476a8085928f3124954b88be712f"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.012326 19902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "b45a4d9f2df04f908d7ff4ef7e2df789"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "08d8ada65ac91d5a1bfe4abf9fd8fbee"
server_key_iv: "ffd6476a8085928f3124954b88be712f"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.013602 19902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:54:18.014269 19917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.014434 19902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:54:18.014530 19902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b45a4d9f2df04f908d7ff4ef7e2df789"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "08d8ada65ac91d5a1bfe4abf9fd8fbee"
server_key_iv: "ffd6476a8085928f3124954b88be712f"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.014616 19902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:18.025437 19902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:18.027133 19902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:18.027289 19902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:18.030886 19902 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:37971
I20260926 14:54:18.030941 19969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:37971 every 8 connection(s)
I20260926 14:54:18.031380 19902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:18.031977 19970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:18.032007 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19902
I20260926 14:54:18.032110 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:54:18.032279 5182 external_mini_cluster.cc:1428] Setting key 22f2878c70e3377031d46095b5f2d1c4
I20260926 14:54:18.035260 19970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap starting.
I20260926 14:54:18.035923 19970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:18.036223 19970 log.cc:824] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:18.036895 19970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: No bootstrap required, opened a new log
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:18.039011 19970 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:18.039158 19970 raft_consensus.cc:374] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:18.039201 19970 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45a4d9f2df04f908d7ff4ef7e2df789, State: Initialized, Role: FOLLOWER
I20260926 14:54:18.039345 19970 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:18.039404 19970 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:18.039426 19970 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:18.039487 19970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:18.040045 19970 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:18.040159 19970 leader_election.cc:302] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45a4d9f2df04f908d7ff4ef7e2df789; no voters:
I20260926 14:54:18.040352 19970 leader_election.cc:288] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:54:18.040381 19975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:18.040481 19975 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 LEADER]: Becoming Leader. State: Replica: b45a4d9f2df04f908d7ff4ef7e2df789, State: Running, Role: LEADER
I20260926 14:54:18.040558 19975 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:18.040604 19970 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:18.040879 19975 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } } }
I20260926 14:54:18.040948 19975 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:18.041059 19975 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45a4d9f2df04f908d7ff4ef7e2df789. Latest consensus state: current_term: 1 leader_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } } }
I20260926 14:54:18.041138 19975 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: This master's current role is: LEADER
W20260926 14:54:18.042546 19991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:54:18.042611 19991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:54:18.042677 19980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:18.042958 19980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:18.043104 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.032753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.5.15.190:37971)
0926 14:54:18.032940 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.032949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.032980 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:54:18.033480 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.033489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.033549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.033693 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:18.034323 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.035695 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.036525 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.036915 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.037031 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.039248 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:18.039252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:18.039253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:18.039265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:18.041275 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.041694 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.041697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.041697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.041794 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.042035 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.042036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.042036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.042383 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:18.042550 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.042680 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.042850 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260926 14:54:18.044373 19980 catalog_manager.cc:1347] Generated new cluster ID: 19e166f0bc62484dbc5a81ed3fe3293f
I20260926 14:54:18.044423 19980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:18.062561 19980 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:54:18.063125 19980 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:18.067663 19980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Generated new TSK 0
I20260926 14:54:18.067845 19980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:54:18.097784 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:37971
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:18.175282 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:18.175457 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:18.175478 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:18.176551 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:18.176587 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:18.176651 19996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:54:18.178035 19996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:18.178268 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:18.178491 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:18.183151 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.183169 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.183162 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.183358 19996 server_base.cc:1034] running on GCE node
I20260926 14:54:18.183561 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:18.183768 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:18.184937 19996 hybrid_clock.cc:648] HybridClock initialized: now 1790434458184927 us; error 26 us; skew 500 ppm
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:18.187433 19996 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:54:18.187870 19996 webserver.cc:460] Webserver started at http://127.5.15.129:34725/ using document root <none> and password file <none>
I20260926 14:54:18.188098 19996 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:18.188163 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:18.188313 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:18.189249 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "789644cf2b6440e3bd7883849cf515ad"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "b3f9069c23f053c0e1a969ee88bee328"
server_key_iv: "8a0811f0deadb8ba2e9c5743a3706b10"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.189564 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "789644cf2b6440e3bd7883849cf515ad"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "b3f9069c23f053c0e1a969ee88bee328"
server_key_iv: "8a0811f0deadb8ba2e9c5743a3706b10"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.190831 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:54:18.191588 20011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.191746 19996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:54:18.191829 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "789644cf2b6440e3bd7883849cf515ad"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "b3f9069c23f053c0e1a969ee88bee328"
server_key_iv: "8a0811f0deadb8ba2e9c5743a3706b10"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.191936 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:18.216150 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:18.218182 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:18.218329 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:18.218557 19996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:18.218902 19996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:18.218962 19996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.219005 19996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:18.219038 19996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.224709 19996 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:41671
I20260926 14:54:18.224754 20124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:41671 every 8 connection(s)
I20260926 14:54:18.225071 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:18.231999 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 19996
I20260926 14:54:18.232098 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:54:18.232281 5182 external_mini_cluster.cc:1428] Setting key 99d32cb609da79eacb8343c4a294c902
I20260926 14:54:18.233847 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.225707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:57467)
0926 14:54:18.225867 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.225899 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.226357 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.226597 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.226600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.227172 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:18.227345 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.227349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.228038 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.228039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.228798 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.228800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.229202 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.229204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.229314 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.229687 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:18.229694 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:18.230985 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:18.232601 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:18.232608 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:18.232610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:18.232820 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:18.233043 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:18.233044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:18.233044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:18.233160 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:18.233430 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:18.233432 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:18.233532 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.233604 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.233711 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260926 14:54:18.233866 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.225732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:57467 (local address 127.5.15.190:37971)
0926 14:54:18.225792 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.225809 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.225972 (+ 163us) server_negotiation.cc:407] Connection header received
0926 14:54:18.226463 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.226464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.226476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.226520 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:18.227445 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.227945 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.228891 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.229112 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.229184 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.231124 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:18.231125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:18.231126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:18.231133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:18.232498 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.232896 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.232897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.232897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.232956 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.233230 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.233231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.233231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.233368 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:18.233443 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.233646 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.233733 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:54:18.234318 20125 heartbeater.cc:347] Connected to a master server at 127.5.15.190:37971
I20260926 14:54:18.234423 20125 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:18.234594 20125 heartbeater.cc:511] Master 127.5.15.190:37971 requested a full tablet report, sending...
I20260926 14:54:18.235102 19934 ts_manager.cc:194] Registered new tserver with Master: 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671)
I20260926 14:54:18.235682 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:57467
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:54:18.257519 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:37971
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:18.335460 20130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:18.335632 20130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:18.335654 20130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:18.336659 20130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:18.336691 20130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:18.336715 20130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:54:18.338053 20130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:18.338207 20130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:18.338357 20130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:18.343317 20136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.343308 20135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.343319 20138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.343559 20130 server_base.cc:1034] running on GCE node
I20260926 14:54:18.343752 20130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:18.343942 20130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:18.345105 20130 hybrid_clock.cc:648] HybridClock initialized: now 1790434458345099 us; error 26 us; skew 500 ppm
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:18.347430 20130 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:54:18.347848 20130 webserver.cc:460] Webserver started at http://127.5.15.130:35249/ using document root <none> and password file <none>
I20260926 14:54:18.348060 20130 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:18.348106 20130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:18.348261 20130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:18.349169 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "e2217900e9424a15865df67e28879b71"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "e1a9dc83a6eb02bdd7807a0cc0480bdf"
server_key_iv: "d533d8b88c6efaff7b468e64472be3e3"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.349485 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "e2217900e9424a15865df67e28879b71"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "e1a9dc83a6eb02bdd7807a0cc0480bdf"
server_key_iv: "d533d8b88c6efaff7b468e64472be3e3"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.350653 20130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:54:18.351346 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.351485 20130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.351581 20130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "e2217900e9424a15865df67e28879b71"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "e1a9dc83a6eb02bdd7807a0cc0480bdf"
server_key_iv: "d533d8b88c6efaff7b468e64472be3e3"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.351631 20130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:18.379292 20130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:18.381469 20130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:18.381572 20130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:18.381753 20130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:18.382000 20130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:18.382030 20130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.382050 20130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:18.382062 20130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.387478 20130 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:46423
I20260926 14:54:18.387562 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:46423 every 8 connection(s)
I20260926 14:54:18.387879 20130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:18.391710 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20130
I20260926 14:54:18.391782 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:54:18.391963 5182 external_mini_cluster.cc:1428] Setting key cb83f6a98cc12897fdaa5026ea6221f5
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:18.397277 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.388613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:57137 (local address 127.5.15.190:37971)
0926 14:54:18.388688 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.388690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.388919 (+ 229us) server_negotiation.cc:407] Connection header received
0926 14:54:18.389431 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.389432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.389443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.389495 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:18.390342 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.390896 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.391974 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.392255 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.392321 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.394619 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:18.394621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:18.394621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:18.394628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:18.395982 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.396367 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.396369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.396370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.396427 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.396675 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.396676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.396676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.396783 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:18.396853 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.397116 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.397189 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:54:18.397447 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.388581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:57137)
0926 14:54:18.388790 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.388822 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.389319 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.389563 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.389567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.390081 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:18.390248 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.390252 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.390982 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.390983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.391892 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.391894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.392492 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.392493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.392662 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.393175 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:18.393183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:18.394535 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:18.396073 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:18.396079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:18.396081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:18.396292 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:18.396491 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:18.396492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:18.396493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:18.396600 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:18.396843 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:18.396849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:18.396977 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.397093 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.397237 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260926 14:54:18.397891 20259 heartbeater.cc:347] Connected to a master server at 127.5.15.190:37971
I20260926 14:54:18.397984 20259 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:18.398213 20259 heartbeater.cc:511] Master 127.5.15.190:37971 requested a full tablet report, sending...
I20260926 14:54:18.398571 19934 ts_manager.cc:194] Registered new tserver with Master: e2217900e9424a15865df67e28879b71 (127.5.15.130:46423)
I20260926 14:54:18.398998 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:57137
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:54:18.417268 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:37971
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:18.496937 20264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:18.497123 20264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:18.497146 20264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:18.498203 20264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:18.498237 20264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:18.498262 20264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:54:18.499658 20264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:18.499832 20264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:18.499990 20264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:18.504762 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.504755 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:18.504755 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:18.504783 20264 server_base.cc:1034] running on GCE node
I20260926 14:54:18.505125 20264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:18.505358 20264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:18.506539 20264 hybrid_clock.cc:648] HybridClock initialized: now 1790434458506524 us; error 28 us; skew 500 ppm
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:18.508853 20264 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:54:18.509258 20264 webserver.cc:460] Webserver started at http://127.5.15.131:38891/ using document root <none> and password file <none>
I20260926 14:54:18.509466 20264 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:18.509531 20264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:18.509655 20264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:54:18.510602 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "be7805aa3891ac3a34a6d3ee817983a1"
server_key_iv: "e80b575dcb0d86297bf6152676c65942"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.510972 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "be7805aa3891ac3a34a6d3ee817983a1"
server_key_iv: "e80b575dcb0d86297bf6152676c65942"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.512315 20264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:18.513036 20279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.513200 20264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:54:18.513299 20264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "be7805aa3891ac3a34a6d3ee817983a1"
server_key_iv: "e80b575dcb0d86297bf6152676c65942"
server_key_version: "encryptionkey@0"
I20260926 14:54:18.513374 20264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:18.535634 20264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:18.537488 20264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:18.537637 20264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:18.537833 20264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:18.538166 20264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:18.538220 20264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.538264 20264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:18.538291 20264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:18.544682 20264 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:40081
I20260926 14:54:18.544719 20392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:40081 every 8 connection(s)
I20260926 14:54:18.545024 20264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:54:18 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:18.551743 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20264
I20260926 14:54:18.551827 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:54:18.552032 5182 external_mini_cluster.cc:1428] Setting key 94522f8012bb86101e8cf9c4ab53a98b
I20260926 14:54:18.554097 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.545692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38397 (local address 127.5.15.190:37971)
0926 14:54:18.545755 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.545757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.545939 (+ 182us) server_negotiation.cc:407] Connection header received
0926 14:54:18.546670 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.546671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.546682 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.546798 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:18.547793 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.548259 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.549170 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.549341 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.549389 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.551235 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:18.551237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:18.551237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:18.551243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:18.552850 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.553268 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.553269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.553269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.553319 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.553510 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.553511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.553511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.553610 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:18.553659 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.553910 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.554003 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:54:18.554240 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.545677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:38397)
0926 14:54:18.545822 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.545861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.546539 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.546896 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.546912 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.547546 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:18.547716 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.547721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.548331 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.548331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.549101 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.549102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.549406 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.549407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.549500 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.549881 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:18.549888 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:18.551156 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:18.552931 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:18.552935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:18.552937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:18.553186 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:18.553360 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:18.553361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:18.553361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:18.553438 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:18.553672 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:18.553677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:18.553808 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.553885 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.554060 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260926 14:54:18.554617 20393 heartbeater.cc:347] Connected to a master server at 127.5.15.190:37971
I20260926 14:54:18.554687 20393 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:18.554888 20393 heartbeater.cc:511] Master 127.5.15.190:37971 requested a full tablet report, sending...
I20260926 14:54:18.555188 19934 ts_manager.cc:194] Registered new tserver with Master: 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081)
I20260926 14:54:18.555544 19934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:38397
I20260926 14:54:18.563733 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:54:18.570055 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.564954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.5.15.190:37971)
0926 14:54:18.565040 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.565042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.565089 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:54:18.565127 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.565128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.565135 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.565180 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:18.565652 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.566083 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.566726 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.566951 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.567000 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.568048 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:18.568050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:18.568050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:18.568057 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:18.569177 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.569465 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.569466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.569466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.569513 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:18.569657 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:18.569657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:18.569658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:18.569773 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:18.569839 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.569918 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.569970 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 14:54:18.571565 19934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:54:18.573078 19934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:54:18.579398 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.575302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41671 (local address 127.0.0.1:48982)
0926 14:54:18.575579 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.575619 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.575691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.575911 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.575913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.575915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:18.575962 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.575964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.577090 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.577091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.578220 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.578222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.579125 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.579196 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.579209 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.579280 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.579329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260926 14:54:18.579406 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.575412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.5.15.129:41671)
0926 14:54:18.575758 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.575762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.575772 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:54:18.575809 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.575811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.575826 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.575890 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:18.576043 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.576998 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.578304 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.579054 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579124 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.579183 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.579294 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.579332 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":69,"threads_started":1}
I20260926 14:54:18.579720 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.575462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:40081 (local address 127.0.0.1:37738)
0926 14:54:18.575996 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.576006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.576058 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.576271 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.576272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.576274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:18.576328 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.576329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.577606 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.577607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.578594 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.578596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.579484 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.579556 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.579560 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.579612 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.579650 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":45,"thread_start_us":67,"threads_started":1}
I20260926 14:54:18.579764 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.575541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.5.15.131:40081)
0926 14:54:18.575722 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.575728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.576115 (+ 387us) server_negotiation.cc:407] Connection header received
0926 14:54:18.576155 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.576158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.576173 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.576249 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:18.576407 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.577503 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.578682 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.579335 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579402 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.579449 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.579639 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.579685 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260926 14:54:18.580211 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.575276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:46423 (local address 127.0.0.1:50502)
0926 14:54:18.575577 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.575620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.575685 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.576548 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.576549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.576551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:18.576590 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.576591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.577774 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.577775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.578972 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.578975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.579958 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.580025 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.580029 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.580118 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.580153 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":45,"threads_started":1}
I20260926 14:54:18.580255 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.575385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50502 (local address 127.5.15.130:46423)
0926 14:54:18.576392 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.576396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.576405 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:54:18.576443 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.576445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.576459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.576538 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:18.576646 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.577672 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579062 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.579786 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.579867 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.579923 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.580138 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.580180 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":84,"threads_started":1}
I20260926 14:54:18.580344 20059 tablet_service.cc:1467] Processing CreateTablet for tablet b227066e935743a4a865ed443cdf9b25 (DEFAULT_TABLE table=test-table [id=4e854f070358488caf87d9f023d60eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:18.580343 20327 tablet_service.cc:1467] Processing CreateTablet for tablet b227066e935743a4a865ed443cdf9b25 (DEFAULT_TABLE table=test-table [id=4e854f070358488caf87d9f023d60eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:18.580665 20059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227066e935743a4a865ed443cdf9b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:18.580665 20327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227066e935743a4a865ed443cdf9b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:18.580804 20193 tablet_service.cc:1467] Processing CreateTablet for tablet b227066e935743a4a865ed443cdf9b25 (DEFAULT_TABLE table=test-table [id=4e854f070358488caf87d9f023d60eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:18.581038 20193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b227066e935743a4a865ed443cdf9b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:18.583455 20410 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Bootstrap starting.
I20260926 14:54:18.583966 20412 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap starting.
I20260926 14:54:18.584131 20410 tablet_bootstrap.cc:654] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:18.584611 20412 tablet_bootstrap.cc:654] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:18.584733 20410 log.cc:824] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:18.585197 20412 log.cc:824] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:18.585528 20410 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: No bootstrap required, opened a new log
I20260926 14:54:18.585529 20411 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Bootstrap starting.
I20260926 14:54:18.585610 20410 ts_tablet_manager.cc:1397] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:18.585901 20412 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: No bootstrap required, opened a new log
I20260926 14:54:18.585979 20412 ts_tablet_manager.cc:1397] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:18.586146 20411 tablet_bootstrap.cc:654] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:18.586675 20411 log.cc:824] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:18.587510 20410 raft_consensus.cc:348] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.587643 20410 raft_consensus.cc:374] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:18.587671 20410 raft_consensus.cc:729] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2217900e9424a15865df67e28879b71, State: Initialized, Role: FOLLOWER
I20260926 14:54:18.587611 20412 raft_consensus.cc:348] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.587824 20412 raft_consensus.cc:374] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:18.587869 20412 raft_consensus.cc:729] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789644cf2b6440e3bd7883849cf515ad, State: Initialized, Role: FOLLOWER
I20260926 14:54:18.587833 20410 consensus_queue.cc:260] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.587980 20412 consensus_queue.cc:260] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.588099 20410 ts_tablet_manager.cc:1428] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 14:54:18.588099 20259 heartbeater.cc:503] Master 127.5.15.190:37971 was elected leader, sending a full tablet report...
I20260926 14:54:18.588250 20412 ts_tablet_manager.cc:1428] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:18.588259 20125 heartbeater.cc:503] Master 127.5.15.190:37971 was elected leader, sending a full tablet report...
I20260926 14:54:18.588742 20411 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: No bootstrap required, opened a new log
I20260926 14:54:18.588816 20411 ts_tablet_manager.cc:1397] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 14:54:18.590296 20411 raft_consensus.cc:348] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.590504 20411 raft_consensus.cc:374] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:18.590551 20411 raft_consensus.cc:729] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eb5a83aadb4a07b9eb5a621d8ecc38, State: Initialized, Role: FOLLOWER
I20260926 14:54:18.590659 20411 consensus_queue.cc:260] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.590940 20411 ts_tablet_manager.cc:1428] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:18.590960 20393 heartbeater.cc:503] Master 127.5.15.190:37971 was elected leader, sending a full tablet report...
W20260926 14:54:18.638433 20260 tablet.cc:2369] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:18.661306 20416 raft_consensus.cc:482] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:18.661410 20416 raft_consensus.cc:504] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.661857 20416 leader_election.cc:288] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081), 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671)
I20260926 14:54:18.664811 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.662022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41671 (local address 127.5.15.130:40015)
0926 14:54:18.662213 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.662222 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.662286 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.662419 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.662420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.662421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:18.662506 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.662507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.663113 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.663114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.664000 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.664002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.664540 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.664541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.664621 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.664626 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.664679 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.664708 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":30,"thread_start_us":110,"threads_started":1}
I20260926 14:54:18.664829 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.662150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:40015 (local address 127.5.15.129:41671)
0926 14:54:18.662196 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.662198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.662307 (+ 109us) server_negotiation.cc:407] Connection header received
0926 14:54:18.662336 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.662337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.662348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.662409 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:18.662577 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.663038 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.664097 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.664467 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.664523 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.664530 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.664703 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.664763 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:54:18.664903 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.662013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:40081 (local address 127.5.15.130:51995)
0926 14:54:18.662088 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.662101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:18.662143 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:18.662322 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:18.662323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:18.662325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:18.662404 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.662406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.663142 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.663143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.663990 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:18.663992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.664617 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.664618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:18.664700 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.664704 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:54:18.664761 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:54:18.664794 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":31}
I20260926 14:54:18.665158 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.662137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:51995 (local address 127.5.15.131:40081)
0926 14:54:18.662188 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.662190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.662199 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:54:18.662229 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.662229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.662241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.662296 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:18.662480 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.663044 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.664074 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.664530 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.664599 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.664609 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.664776 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.665095 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 14:54:18.665138 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227066e935743a4a865ed443cdf9b25" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789644cf2b6440e3bd7883849cf515ad" is_pre_election: true
I20260926 14:54:18.665294 20079 raft_consensus.cc:2454] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 0.
I20260926 14:54:18.665464 20347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227066e935743a4a865ed443cdf9b25" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" is_pre_election: true
I20260926 14:54:18.665570 20149 leader_election.cc:302] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789644cf2b6440e3bd7883849cf515ad, e2217900e9424a15865df67e28879b71; no voters:
I20260926 14:54:18.665572 20347 raft_consensus.cc:2454] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 0.
I20260926 14:54:18.665683 20416 raft_consensus.cc:2781] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:54:18.665737 20416 raft_consensus.cc:482] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:54:18.665764 20416 raft_consensus.cc:3037] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:18.666452 20416 raft_consensus.cc:504] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.666589 20416 leader_election.cc:288] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 election: Requested vote from peers 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081), 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671)
I20260926 14:54:18.666734 20079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227066e935743a4a865ed443cdf9b25" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789644cf2b6440e3bd7883849cf515ad"
I20260926 14:54:18.666818 20347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b227066e935743a4a865ed443cdf9b25" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
I20260926 14:54:18.666848 20079 raft_consensus.cc:3037] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:18.666880 20347 raft_consensus.cc:3037] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:18.667693 20079 raft_consensus.cc:2454] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 1.
I20260926 14:54:18.667693 20347 raft_consensus.cc:2454] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 1.
I20260926 14:54:18.667877 20149 leader_election.cc:302] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 789644cf2b6440e3bd7883849cf515ad, e2217900e9424a15865df67e28879b71; no voters:
I20260926 14:54:18.668011 20416 raft_consensus.cc:2781] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:18.668184 20416 raft_consensus.cc:686] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 1 LEADER]: Becoming Leader. State: Replica: e2217900e9424a15865df67e28879b71, State: Running, Role: LEADER
I20260926 14:54:18.668301 20416 consensus_queue.cc:237] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:18.669168 19933 catalog_manager.cc:5495] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2217900e9424a15865df67e28879b71 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "e2217900e9424a15865df67e28879b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } health_report { overall_health: UNKNOWN } } }
I20260926 14:54:18.697009 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.694671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.5.15.130:46423)
0926 14:54:18.694708 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:18.694710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:18.694780 (+ 70us) server_negotiation.cc:407] Connection header received
0926 14:54:18.694855 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:18.694856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:18.694872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:18.694931 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:18.695113 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.695542 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.696287 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:18.696465 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:18.696534 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:18.696700 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:18.696789 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:18.696878 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:18.696897 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 14:54:18.696915 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:54:18.698611 20347 raft_consensus.cc:1261] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Refusing update from remote peer e2217900e9424a15865df67e28879b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:18.698668 20079 raft_consensus.cc:1261] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Refusing update from remote peer e2217900e9424a15865df67e28879b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:18.699019 20416 consensus_queue.cc:1035] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:18.699141 20420 consensus_queue.cc:1035] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:18.701570 20424 mvcc.cc:205] Tried to move back new op lower bound from 7333619542826999808 to 7333619542705922048. Current Snapshot: MvccSnapshot[applied={T|T < 7333619542826999808}]
I20260926 14:54:18.701644 20425 mvcc.cc:205] Tried to move back new op lower bound from 7333619542826999808 to 7333619542705922048. Current Snapshot: MvccSnapshot[applied={T|T < 7333619542826999808}]
I20260926 14:54:18.703367 19934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:18.703469 19934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:18.704154 19934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Sending DeleteTablet for 3 replicas of tablet b227066e935743a4a865ed443cdf9b25
I20260926 14:54:18.704450 20193 tablet_service.cc:1514] Processing DeleteTablet for tablet b227066e935743a4a865ed443cdf9b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:50502
I20260926 14:54:18.704494 20059 tablet_service.cc:1514] Processing DeleteTablet for tablet b227066e935743a4a865ed443cdf9b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:48982
I20260926 14:54:18.704532 20327 tablet_service.cc:1514] Processing DeleteTablet for tablet b227066e935743a4a865ed443cdf9b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:37738
I20260926 14:54:18.704598 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19996
I20260926 14:54:18.705145 20433 tablet_replica.cc:330] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: stopping tablet replica
I20260926 14:54:18.705255 20433 raft_consensus.cc:2229] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:54:18.705368 20433 raft_consensus.cc:2258] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:18.705713 20433 ts_tablet_manager.cc:1905] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:18.707803 20433 ts_tablet_manager.cc:1918] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:18.707882 20433 log.cc:1196] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/b227066e935743a4a865ed443cdf9b25
I20260926 14:54:18.708145 20433 ts_tablet_manager.cc:1939] T b227066e935743a4a865ed443cdf9b25 P e2217900e9424a15865df67e28879b71: Deleting consensus metadata
I20260926 14:54:18.708544 19920 catalog_manager.cc:4841] TS e2217900e9424a15865df67e28879b71 (127.5.15.130:46423): tablet b227066e935743a4a865ed443cdf9b25 (table test-table [id=4e854f070358488caf87d9f023d60eae]) successfully deleted
I20260926 14:54:18.711126 20434 tablet_replica.cc:330] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: stopping tablet replica
I20260926 14:54:18.711205 20434 raft_consensus.cc:2229] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:18.711328 20434 raft_consensus.cc:2258] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:54:18.711573 19921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:41671 (error 108)
I20260926 14:54:18.711758 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 20130
I20260926 14:54:18.712407 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:18.711917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41671 (local address 127.0.0.1:48994)
0926 14:54:18.712333 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:18.712371 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
W20260926 14:54:18.712500 19921 catalog_manager.cc:4584] TS 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b227066e935743a4a865ed443cdf9b25: Network error: Client connection negotiation failed: client connection to 127.5.15.129:41671: connect: Connection refused (error 111)
I20260926 14:54:18.713470 20434 ts_tablet_manager.cc:1905] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:18.714877 20434 ts_tablet_manager.cc:1918] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:18.714955 20434 log.cc:1196] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/b227066e935743a4a865ed443cdf9b25
I20260926 14:54:18.715196 20434 ts_tablet_manager.cc:1939] T b227066e935743a4a865ed443cdf9b25 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Deleting consensus metadata
I20260926 14:54:18.715588 19920 catalog_manager.cc:4841] TS 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081): tablet b227066e935743a4a865ed443cdf9b25 (table test-table [id=4e854f070358488caf87d9f023d60eae]) successfully deleted
I20260926 14:54:18.717113 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 20264
I20260926 14:54:18.721691 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 19902
I20260926 14:54:38.726902 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37971
--webserver_interface=127.5.15.190
--webserver_port=37471
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:38.805801 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:38.805982 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:38.806003 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:38.807164 20437 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:54:38.807202 20437 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:54:38.807215 20437 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:54:38.807227 20437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:38.807274 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:38.807302 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:38.807324 20437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:38.807343 20437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:38.808811 20437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:37971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:38.809010 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:38.809227 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:54:38.817950 20437 server_base.cc:1034] running on GCE node
W20260926 14:54:38.818058 20446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:38.818101 20443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:38.818022 20444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:38.818483 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:38.818853 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:38.820019 20437 hybrid_clock.cc:648] HybridClock initialized: now 1790434478820007 us; error 43 us; skew 500 ppm
Sep 26 14:54:38 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:38.822372 20437 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:54:38.822729 20437 webserver.cc:460] Webserver started at http://127.5.15.190:37471/ using document root <none> and password file <none>
I20260926 14:54:38.822988 20437 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:38.823040 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:38.824298 20437 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:54:38.824898 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:38.825109 20437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:54:38.825189 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b45a4d9f2df04f908d7ff4ef7e2df789"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "08d8ada65ac91d5a1bfe4abf9fd8fbee"
server_key_iv: "ffd6476a8085928f3124954b88be712f"
server_key_version: "encryptionkey@0"
I20260926 14:54:38.825515 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:38.841714 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:38.843883 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:38.843982 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:38.847667 20437 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:37971
I20260926 14:54:38.847738 20505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:37971 every 8 connection(s)
I20260926 14:54:38.848096 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:38.849542 20506 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:54:38.850191 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap starting.
I20260926 14:54:38.852234 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20437
I20260926 14:54:38.852506 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:41671
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:37971
--webserver_port=34725
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:38.859069 20506 log.cc:824] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:38.860846 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:38.861140 20506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap complete.
I20260926 14:54:38.863472 20506 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:38.863813 20506 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45a4d9f2df04f908d7ff4ef7e2df789, State: Initialized, Role: FOLLOWER
I20260926 14:54:38.864017 20506 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:38.864108 20506 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:38.864171 20506 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:38.864238 20506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:54:38.865299 20506 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:38.865427 20506 leader_election.cc:302] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45a4d9f2df04f908d7ff4ef7e2df789; no voters:
I20260926 14:54:38.865620 20506 leader_election.cc:288] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:54:38.865653 20510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:54:38.865767 20510 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 LEADER]: Becoming Leader. State: Replica: b45a4d9f2df04f908d7ff4ef7e2df789, State: Running, Role: LEADER
I20260926 14:54:38.865887 20506 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:38.865924 20510 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:38.866242 20510 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45a4d9f2df04f908d7ff4ef7e2df789. Latest consensus state: current_term: 2 leader_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } } }
I20260926 14:54:38.866392 20510 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:38.866245 20511 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } } }
I20260926 14:54:38.866665 20511 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:38.866986 20518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:38.867813 20518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:38.867914 20518 catalog_manager.cc:1259] Loaded cluster ID: 19e166f0bc62484dbc5a81ed3fe3293f
I20260926 14:54:38.867944 20518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:38.869100 20518 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:38.869274 20518 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Loaded TSK: 0
I20260926 14:54:38.883991 20518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Generated new TSK 1
I20260926 14:54:38.884086 20518 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:54:38.934190 20508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:38.934432 20508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:38.934469 20508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:38.935662 20508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:38.935724 20508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:38.935768 20508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:54:38.937216 20508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=34725
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:38.937461 20508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:38.937682 20508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:38.943490 20534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:38.943588 20533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:38.943571 20508 server_base.cc:1034] running on GCE node
W20260926 14:54:38.943500 20536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:38.943872 20508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:38.944142 20508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:38.945302 20508 hybrid_clock.cc:648] HybridClock initialized: now 1790434478945299 us; error 36 us; skew 500 ppm
Sep 26 14:54:38 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:38.948089 20508 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:54:38.948578 20508 webserver.cc:460] Webserver started at http://127.5.15.129:34725/ using document root <none> and password file <none>
I20260926 14:54:38.948853 20508 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:38.948935 20508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:38.950630 20508 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:54:38.951423 20543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:38.951632 20508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:54:38.951726 20508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "789644cf2b6440e3bd7883849cf515ad"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "b3f9069c23f053c0e1a969ee88bee328"
server_key_iv: "8a0811f0deadb8ba2e9c5743a3706b10"
server_key_version: "encryptionkey@0"
I20260926 14:54:38.952143 20508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:38.987478 20508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:38.989717 20508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:38.989862 20508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:38.990160 20508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:38.990595 20550 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:54:38.992041 20508 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:54:38.992100 20508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 14:54:38.992131 20508 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:54:38.992957 20508 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:54:38.992998 20508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:38.993093 20550 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap starting.
I20260926 14:54:38.999734 20508 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:41671
I20260926 14:54:38.999971 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:41671 every 8 connection(s)
I20260926 14:54:39.000155 20508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:54:39.003173 20550 log.cc:824] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:39.005633 20550 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:39.005954 20550 tablet_bootstrap.cc:492] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap complete.
I20260926 14:54:39.007115 20550 ts_tablet_manager.cc:1397] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
I20260926 14:54:39.007958 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20508
I20260926 14:54:39.008222 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:46423
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:37971
--webserver_port=35249
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:39.008435 20550 raft_consensus.cc:348] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.008709 20550 raft_consensus.cc:729] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789644cf2b6440e3bd7883849cf515ad, State: Initialized, Role: FOLLOWER
I20260926 14:54:39.008841 20550 consensus_queue.cc:260] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.009078 20550 ts_tablet_manager.cc:1428] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 26 14:54:39 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:39.015136 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.000922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:41935 (local address 127.5.15.190:37971)
0926 14:54:39.001142 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.001153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.003051 (+ 1898us) server_negotiation.cc:407] Connection header received
0926 14:54:39.003712 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.003719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.003774 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.003927 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:39.004547 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.005400 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.006606 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.007065 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.007186 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.011628 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:39.011633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:39.011633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:39.011644 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:39.013306 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.013736 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.013739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.013739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.013885 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.014234 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.014235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.014235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.014442 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:39.014505 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.014903 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.015007 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260926 14:54:39.015383 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.000855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:41935)
0926 14:54:39.002869 (+ 2014us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.002915 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.003493 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.003970 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.003973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.004255 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:39.004429 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.004434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.005524 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.005525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.006498 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.006501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.007151 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.007152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.007289 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.009701 (+ 2412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:39.009712 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:39.011517 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:39.013425 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:39.013432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:39.013434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:39.013652 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:39.013974 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:39.013975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:39.013976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:39.014138 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:39.014513 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:39.014520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:39.014696 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.014855 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.015044 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1970,"thread_start_us":105,"threads_started":1}
I20260926 14:54:39.016135 20658 heartbeater.cc:347] Connected to a master server at 127.5.15.190:37971
I20260926 14:54:39.016243 20658 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:39.016434 20658 heartbeater.cc:511] Master 127.5.15.190:37971 requested a full tablet report, sending...
I20260926 14:54:39.017148 20470 ts_manager.cc:194] Registered new tserver with Master: 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671)
I20260926 14:54:39.018096 20470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:41935
I20260926 14:54:39.019124 20658 heartbeater.cc:503] Master 127.5.15.190:37971 was elected leader, sending a full tablet report...
I20260926 14:54:39.026130 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.017790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.5.15.129:41671)
0926 14:54:39.022866 (+ 5076us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.022870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.022954 (+ 84us) server_negotiation.cc:407] Connection header received
0926 14:54:39.023037 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.023039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.023051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.023117 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:39.023259 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.023880 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.025389 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.025808 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.025863 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.025896 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.026058 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.026086 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5051,"thread_start_us":77,"threads_started":1}
I20260926 14:54:39.026162 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.017726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41671 (local address 127.0.0.1:41808)
0926 14:54:39.022849 (+ 5123us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.022889 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.022961 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.023138 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.023140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.023142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:39.023198 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.023200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.023948 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.023949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.025303 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.025306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.025883 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.025884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.025961 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.025974 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.026041 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.026089 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5092,"thread_start_us":81,"threads_started":1}
I20260926 14:54:39.026665 20592 tablet_service.cc:1514] Processing DeleteTablet for tablet b227066e935743a4a865ed443cdf9b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:41808
I20260926 14:54:39.027253 20667 tablet_replica.cc:330] stopping tablet replica
I20260926 14:54:39.027369 20667 raft_consensus.cc:2229] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:39.027433 20667 raft_consensus.cc:2258] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:39.027720 20667 ts_tablet_manager.cc:1905] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:39.029848 20667 ts_tablet_manager.cc:1918] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:39.029954 20667 log.cc:1196] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal/wals/b227066e935743a4a865ed443cdf9b25
I20260926 14:54:39.030175 20667 ts_tablet_manager.cc:1939] T b227066e935743a4a865ed443cdf9b25 P 789644cf2b6440e3bd7883849cf515ad: Deleting consensus metadata
I20260926 14:54:39.030629 20457 catalog_manager.cc:4841] TS 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671): tablet b227066e935743a4a865ed443cdf9b25 (table test-table [id=4e854f070358488caf87d9f023d60eae]) successfully deleted
W20260926 14:54:39.093514 20664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:39.093698 20664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:39.093720 20664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:39.094856 20664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:39.094894 20664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:39.094920 20664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:54:39.096293 20664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:46423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=35249
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:39.096474 20664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:39.096640 20664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:39.101544 20672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:39.101557 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:39.101557 20673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:39.101842 20664 server_base.cc:1034] running on GCE node
I20260926 14:54:39.102036 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:39.102269 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:39.103441 20664 hybrid_clock.cc:648] HybridClock initialized: now 1790434479103427 us; error 32 us; skew 500 ppm
Sep 26 14:54:39 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:39.105819 20664 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:54:39.106249 20664 webserver.cc:460] Webserver started at http://127.5.15.130:35249/ using document root <none> and password file <none>
I20260926 14:54:39.106482 20664 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:39.106562 20664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:39.107918 20664 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:39.108451 20682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.108623 20664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:54:39.108718 20664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "e2217900e9424a15865df67e28879b71"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "e1a9dc83a6eb02bdd7807a0cc0480bdf"
server_key_iv: "d533d8b88c6efaff7b468e64472be3e3"
server_key_version: "encryptionkey@0"
I20260926 14:54:39.109045 20664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:39.119593 20664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:39.121291 20664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:39.121454 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:39.121670 20664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:39.121969 20664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:39.122020 20664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.122058 20664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:39.122084 20664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.128012 20664 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:46423
I20260926 14:54:39.128057 20795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:46423 every 8 connection(s)
I20260926 14:54:39.128419 20664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:39.133451 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20664
I20260926 14:54:39.133718 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:40081
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:37971
--webserver_port=38891
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:54:39 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:39.137548 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.129066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:56777 (local address 127.5.15.190:37971)
0926 14:54:39.129111 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.129113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.129331 (+ 218us) server_negotiation.cc:407] Connection header received
0926 14:54:39.129790 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.129791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.129819 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.129870 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:39.130814 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.131287 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.132191 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.132374 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.132422 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.134552 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:39.134553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:39.134554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:39.134561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:39.136169 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.136640 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.136641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.136641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.136693 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.136960 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.136961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.136961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.137056 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:39.137122 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.137421 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.137488 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:54:39.137825 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.129069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:56777)
0926 14:54:39.129213 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.129247 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.129694 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.129921 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.129924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.130474 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:39.130679 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.130684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.131363 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.131363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.132119 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.132121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.132441 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.132442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.132537 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.132951 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:39.132958 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:39.134455 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:39.136255 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:39.136261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:39.136263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:39.136560 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:39.136761 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:39.136762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:39.136763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:39.136900 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:39.137127 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:39.137132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:39.137293 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.137402 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.137573 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 14:54:39.138442 20796 heartbeater.cc:347] Connected to a master server at 127.5.15.190:37971
I20260926 14:54:39.138527 20796 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:39.138710 20796 heartbeater.cc:511] Master 127.5.15.190:37971 requested a full tablet report, sending...
I20260926 14:54:39.139165 20470 ts_manager.cc:194] Registered new tserver with Master: e2217900e9424a15865df67e28879b71 (127.5.15.130:46423)
I20260926 14:54:39.139551 20470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:56777
W20260926 14:54:39.215886 20799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:39.216101 20799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:39.216140 20799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:39.217268 20799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:39.217339 20799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:39.217383 20799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:54:39.218892 20799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:40081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=38891
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:39.219148 20799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:39.219376 20799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:39.224229 20804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:39.224222 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:39.224205 20805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:39.224363 20799 server_base.cc:1034] running on GCE node
I20260926 14:54:39.224581 20799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:39.224761 20799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:39.225925 20799 hybrid_clock.cc:648] HybridClock initialized: now 1790434479225908 us; error 32 us; skew 500 ppm
Sep 26 14:54:39 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:39.228225 20799 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:54:39.228664 20799 webserver.cc:460] Webserver started at http://127.5.15.131:38891/ using document root <none> and password file <none>
I20260926 14:54:39.228915 20799 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:39.228994 20799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:39.230252 20799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:39.230782 20814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.230957 20799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.231042 20799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "be7805aa3891ac3a34a6d3ee817983a1"
server_key_iv: "e80b575dcb0d86297bf6152676c65942"
server_key_version: "encryptionkey@0"
I20260926 14:54:39.231343 20799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:39.239517 20799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:39.241182 20799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:39.241331 20799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:39.241539 20799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:39.241823 20799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:39.241871 20799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.241909 20799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:39.241932 20799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.247433 20799 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:40081
I20260926 14:54:39.247489 20927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:40081 every 8 connection(s)
I20260926 14:54:39.247798 20799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:54:39.249118 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20799
Sep 26 14:54:39 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:54:39.256570 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.249547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.5.15.190:37971)
0926 14:54:39.249648 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.249649 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.249791 (+ 142us) server_negotiation.cc:407] Connection header received
0926 14:54:39.249870 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.249871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.249882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.249960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:39.250575 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.251173 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.251948 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.252256 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.252310 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.253548 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:39.253550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:39.253550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:39.253558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:39.255118 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.255491 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.255492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.255492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.255544 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.255751 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.255752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.255752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.255892 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:39.256026 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.256292 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.256425 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":83,"thread_start_us":66,"threads_started":1}
I20260926 14:54:39.258181 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.248632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:56961 (local address 127.5.15.190:37971)
0926 14:54:39.248712 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.248715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.248874 (+ 159us) server_negotiation.cc:407] Connection header received
0926 14:54:39.249486 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.249488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.249501 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.249567 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:39.250528 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.251149 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.252219 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.252470 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.252542 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.254988 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:39.254989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:39.254990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:39.254996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:39.256675 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.257229 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.257230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.257230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.257285 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.257561 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.257562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.257562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.257663 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:39.257729 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.258004 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.258093 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":48}
I20260926 14:54:39.258383 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.248601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:56961)
0926 14:54:39.248762 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.248795 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.249366 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.249640 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.249644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.250250 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:54:39.250433 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.250437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.251241 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.251243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.252140 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.252142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.252607 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.252608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.252750 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.253251 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:54:39.253258 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:54:39.254880 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:54:39.256818 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:39.256824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:39.256826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:39.257149 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:39.257372 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:54:39.257372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:54:39.257373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:54:39.257479 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:54:39.257732 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:54:39.257738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:54:39.257871 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.257983 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.258152 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260926 14:54:39.258826 20928 heartbeater.cc:347] Connected to a master server at 127.5.15.190:37971
I20260926 14:54:39.258939 20928 heartbeater.cc:464] Registering TS with master...
I20260926 14:54:39.259106 20928 heartbeater.cc:511] Master 127.5.15.190:37971 requested a full tablet report, sending...
I20260926 14:54:39.259446 20470 ts_manager.cc:194] Registered new tserver with Master: 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081)
I20260926 14:54:39.259795 20470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:56961
I20260926 14:54:39.267962 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:54:39.270802 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.268291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.5.15.190:37971)
0926 14:54:39.268347 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.268349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.268460 (+ 111us) server_negotiation.cc:407] Connection header received
0926 14:54:39.268543 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.268544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.268598 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.268667 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:39.268808 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.269248 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.269989 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.270175 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.270245 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.270434 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:39.270495 (+ 61us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 14:54:39.270666 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47684: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 14:54:39.270855 20661 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47684: token expired
W20260926 14:54:39.270887 20396 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.15.190:37971)
I20260926 14:54:39.276434 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.271199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.5.15.190:37971)
0926 14:54:39.271252 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.271254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.271289 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:54:39.271350 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.271351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.271360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.271407 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:54:39.271856 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.272293 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.272939 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.273114 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.273183 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.274165 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:54:39.274167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:54:39.274167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:54:39.274174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:54:39.275346 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.275703 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.275704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.275704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.275758 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:54:39.275957 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:54:39.275958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:54:39.275959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:54:39.276079 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:54:39.276180 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.276280 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.276349 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:54:39.277434 20470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:54:39.279089 20470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:54:39.281270 20592 tablet_service.cc:1467] Processing CreateTablet for tablet 34104a3f89d44cd9a6261629cea906b2 (DEFAULT_TABLE table=test-table [id=52cc1f02ceec414685966f98c6963afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:39.281482 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34104a3f89d44cd9a6261629cea906b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:39.283008 20550 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap starting.
I20260926 14:54:39.283317 20550 tablet_bootstrap.cc:654] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:39.283836 20550 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: No bootstrap required, opened a new log
I20260926 14:54:39.283882 20550 ts_tablet_manager.cc:1397] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:39.284008 20550 raft_consensus.cc:348] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.284080 20550 raft_consensus.cc:374] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:39.284109 20550 raft_consensus.cc:729] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789644cf2b6440e3bd7883849cf515ad, State: Initialized, Role: FOLLOWER
I20260926 14:54:39.284168 20550 consensus_queue.cc:260] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.284260 20550 ts_tablet_manager.cc:1428] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:39.284680 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.280862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:46423 (local address 127.0.0.1:36812)
0926 14:54:39.280897 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.280908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.280941 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.281321 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.281322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.281323 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:39.281362 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.281364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.282396 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.282397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.283490 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.283492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.284473 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.284474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.284540 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.284544 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.284574 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.284614 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 14:54:39.284960 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.281036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.5.15.131:40081)
0926 14:54:39.281194 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.281198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.281360 (+ 162us) server_negotiation.cc:407] Connection header received
0926 14:54:39.281400 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.281402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.281418 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.281492 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:39.281574 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.282551 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.283819 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.284496 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.284561 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.284619 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.284837 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.284886 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260926 14:54:39.284997 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.280962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.5.15.130:46423)
0926 14:54:39.281161 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.281165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.281175 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:54:39.281211 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.281213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.281234 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.281322 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:39.281413 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.282321 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.283580 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.284374 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.284792 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.284860 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.284883 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.284917 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260926 14:54:39.284986 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.280953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:40081 (local address 127.0.0.1:48682)
0926 14:54:39.281244 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.281253 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.281308 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.281486 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.281486 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.281488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:39.281515 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.281516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.282634 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.282635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.283712 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.283714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.284561 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.284562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.284642 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.284647 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.284700 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.284743 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260926 14:54:39.285633 20730 tablet_service.cc:1467] Processing CreateTablet for tablet 34104a3f89d44cd9a6261629cea906b2 (DEFAULT_TABLE table=test-table [id=52cc1f02ceec414685966f98c6963afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:39.285964 20730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34104a3f89d44cd9a6261629cea906b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:39.287129 20862 tablet_service.cc:1467] Processing CreateTablet for tablet 34104a3f89d44cd9a6261629cea906b2 (DEFAULT_TABLE table=test-table [id=52cc1f02ceec414685966f98c6963afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:54:39.287420 20862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34104a3f89d44cd9a6261629cea906b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:54:39.288825 20938 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Bootstrap starting.
I20260926 14:54:39.289489 20938 tablet_bootstrap.cc:654] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:39.290078 20938 log.cc:824] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:39.290390 20939 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Bootstrap starting.
I20260926 14:54:39.291011 20938 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: No bootstrap required, opened a new log
I20260926 14:54:39.291095 20938 ts_tablet_manager.cc:1397] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:39.291111 20939 tablet_bootstrap.cc:654] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Neither blocks nor log segments found. Creating new log.
I20260926 14:54:39.291566 20939 log.cc:824] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:39.292250 20939 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: No bootstrap required, opened a new log
I20260926 14:54:39.292338 20939 ts_tablet_manager.cc:1397] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:54:39.292820 20938 raft_consensus.cc:348] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.292950 20938 raft_consensus.cc:374] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:39.292970 20938 raft_consensus.cc:729] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2217900e9424a15865df67e28879b71, State: Initialized, Role: FOLLOWER
I20260926 14:54:39.293099 20938 consensus_queue.cc:260] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.293363 20938 ts_tablet_manager.cc:1428] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:54:39.293387 20796 heartbeater.cc:503] Master 127.5.15.190:37971 was elected leader, sending a full tablet report...
I20260926 14:54:39.294412 20939 raft_consensus.cc:348] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.294559 20939 raft_consensus.cc:374] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:54:39.294581 20939 raft_consensus.cc:729] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21eb5a83aadb4a07b9eb5a621d8ecc38, State: Initialized, Role: FOLLOWER
I20260926 14:54:39.294691 20939 consensus_queue.cc:260] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.294986 20939 ts_tablet_manager.cc:1428] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 14:54:39.294996 20928 heartbeater.cc:503] Master 127.5.15.190:37971 was elected leader, sending a full tablet report...
W20260926 14:54:39.378953 20797 tablet.cc:2369] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:54:39.407753 20942 raft_consensus.cc:482] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:54:39.407905 20942 raft_consensus.cc:504] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.408378 20942 leader_election.cc:288] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081), 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671)
I20260926 14:54:39.412379 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.408669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:39653 (local address 127.5.15.129:41671)
0926 14:54:39.408745 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.408748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.408844 (+ 96us) server_negotiation.cc:407] Connection header received
0926 14:54:39.408879 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.408880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.408892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.408951 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:39.409195 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.409897 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.411424 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.411999 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.412074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.412083 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.412254 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.412298 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:54:39.412369 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.408754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:46533 (local address 127.5.15.131:40081)
0926 14:54:39.408858 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.408860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.408869 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:54:39.408893 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.408894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.408903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.408954 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:54:39.409149 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.409891 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.411424 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.411997 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.412074 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.412083 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.412254 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.412303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 14:54:39.412395 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.408565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:40081 (local address 127.5.15.130:46533)
0926 14:54:39.408614 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.408627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.408677 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.408975 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.408976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.408978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:39.409075 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.409077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.409976 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.409976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.411330 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.411333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.412080 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.412081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.412162 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.412169 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.412230 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.412283 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":75}
I20260926 14:54:39.412400 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.408578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41671 (local address 127.5.15.130:39653)
0926 14:54:39.408756 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.408767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:39.408832 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:39.408975 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:54:39.408976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:54:39.408978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:54:39.409117 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.409119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.409976 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.409976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.411331 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:54:39.411333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.412100 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.412100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:54:39.412162 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.412169 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:54:39.412230 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:54:39.412283 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":77,"thread_start_us":101,"threads_started":1}
I20260926 14:54:39.412788 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34104a3f89d44cd9a6261629cea906b2" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" is_pre_election: true
I20260926 14:54:39.412986 20882 raft_consensus.cc:2454] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 0.
I20260926 14:54:39.413218 20685 leader_election.cc:302] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eb5a83aadb4a07b9eb5a621d8ecc38, e2217900e9424a15865df67e28879b71; no voters:
I20260926 14:54:39.413388 20942 raft_consensus.cc:2781] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:54:39.413491 20942 raft_consensus.cc:482] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:54:39.413529 20942 raft_consensus.cc:3037] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:39.412935 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34104a3f89d44cd9a6261629cea906b2" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789644cf2b6440e3bd7883849cf515ad" is_pre_election: true
I20260926 14:54:39.413647 20612 raft_consensus.cc:2454] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 0.
I20260926 14:54:39.414408 20942 raft_consensus.cc:504] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.414559 20942 leader_election.cc:288] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 election: Requested vote from peers 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081), 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671)
I20260926 14:54:39.414686 20882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34104a3f89d44cd9a6261629cea906b2" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
I20260926 14:54:39.414806 20882 raft_consensus.cc:3037] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:39.415493 20882 raft_consensus.cc:2454] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 1.
I20260926 14:54:39.415611 20612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34104a3f89d44cd9a6261629cea906b2" candidate_uuid: "e2217900e9424a15865df67e28879b71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789644cf2b6440e3bd7883849cf515ad"
I20260926 14:54:39.415639 20685 leader_election.cc:302] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21eb5a83aadb4a07b9eb5a621d8ecc38, e2217900e9424a15865df67e28879b71; no voters:
I20260926 14:54:39.415688 20612 raft_consensus.cc:3037] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:54:39.415776 20942 raft_consensus.cc:2781] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:54:39.415946 20942 raft_consensus.cc:686] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 1 LEADER]: Becoming Leader. State: Replica: e2217900e9424a15865df67e28879b71, State: Running, Role: LEADER
I20260926 14:54:39.416038 20942 consensus_queue.cc:237] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:39.416558 20612 raft_consensus.cc:2454] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2217900e9424a15865df67e28879b71 in term 1.
I20260926 14:54:39.416882 20470 catalog_manager.cc:5495] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2217900e9424a15865df67e28879b71 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "e2217900e9424a15865df67e28879b71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } health_report { overall_health: UNKNOWN } } }
I20260926 14:54:39.428297 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.426242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.5.15.130:46423)
0926 14:54:39.426295 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 14:54:39.426296 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:54:39.426303 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:54:39.426368 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:54:39.426369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:54:39.426382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:54:39.426440 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:54:39.426544 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.427043 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.427703 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:54:39.427869 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:54:39.427916 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:54:39.428048 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:54:39.428140 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:54:39.428197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:54:39.428217 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 14:54:39.428232 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:54:39.429911 20882 raft_consensus.cc:1261] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Refusing update from remote peer e2217900e9424a15865df67e28879b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:39.429911 20612 raft_consensus.cc:1261] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Refusing update from remote peer e2217900e9424a15865df67e28879b71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:54:39.430300 20945 consensus_queue.cc:1035] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:39.430488 20942 consensus_queue.cc:1035] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:54:39.433003 20950 mvcc.cc:205] Tried to move back new op lower bound from 7333619627742449664 to 7333619627688574976. Current Snapshot: MvccSnapshot[applied={T|T < 7333619627742449664}]
I20260926 14:54:39.434883 20469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:39.434978 20469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:54:39.435678 20469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Sending DeleteTablet for 3 replicas of tablet 34104a3f89d44cd9a6261629cea906b2
I20260926 14:54:39.436000 20592 tablet_service.cc:1514] Processing DeleteTablet for tablet 34104a3f89d44cd9a6261629cea906b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:39 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:41808
I20260926 14:54:39.436028 20862 tablet_service.cc:1514] Processing DeleteTablet for tablet 34104a3f89d44cd9a6261629cea906b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:39 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:48682
I20260926 14:54:39.436046 20730 tablet_service.cc:1514] Processing DeleteTablet for tablet 34104a3f89d44cd9a6261629cea906b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:54:39 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:36812
I20260926 14:54:39.436090 20667 tablet_replica.cc:330] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: stopping tablet replica
I20260926 14:54:39.436136 20667 raft_consensus.cc:2229] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:39.436193 20667 raft_consensus.cc:2258] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:39.436312 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 20508
I20260926 14:54:39.436590 20667 ts_tablet_manager.cc:1905] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:39.437281 20958 tablet_replica.cc:330] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: stopping tablet replica
I20260926 14:54:39.437377 20958 raft_consensus.cc:2229] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:54:39.437494 20958 raft_consensus.cc:2258] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:39.437932 20958 ts_tablet_manager.cc:1905] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:39.439935 20958 ts_tablet_manager.cc:1918] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:39.440008 20958 log.cc:1196] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/34104a3f89d44cd9a6261629cea906b2
I20260926 14:54:39.440297 20958 ts_tablet_manager.cc:1939] T 34104a3f89d44cd9a6261629cea906b2 P e2217900e9424a15865df67e28879b71: Deleting consensus metadata
I20260926 14:54:39.440686 20456 catalog_manager.cc:4841] TS e2217900e9424a15865df67e28879b71 (127.5.15.130:46423): tablet 34104a3f89d44cd9a6261629cea906b2 (table test-table [id=52cc1f02ceec414685966f98c6963afe]) successfully deleted
W20260926 14:54:39.443060 20457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:41671 (error 108)
I20260926 14:54:39.443291 20957 tablet_replica.cc:330] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: stopping tablet replica
I20260926 14:54:39.443408 20957 raft_consensus.cc:2229] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:54:39.443513 20957 raft_consensus.cc:2258] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:54:39.443672 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:39.443553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:41671 (local address 127.0.0.1:41822)
0926 14:54:39.443598 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:39.443642 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 14:54:39.443689 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 20664
W20260926 14:54:39.444378 20457 catalog_manager.cc:4584] TS 789644cf2b6440e3bd7883849cf515ad (127.5.15.129:41671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34104a3f89d44cd9a6261629cea906b2: Network error: Client connection negotiation failed: client connection to 127.5.15.129:41671: connect: Connection refused (error 111)
I20260926 14:54:39.445089 20957 ts_tablet_manager.cc:1905] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:54:39.446327 20957 ts_tablet_manager.cc:1918] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:54:39.446417 20957 log.cc:1196] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/34104a3f89d44cd9a6261629cea906b2
I20260926 14:54:39.446671 20957 ts_tablet_manager.cc:1939] T 34104a3f89d44cd9a6261629cea906b2 P 21eb5a83aadb4a07b9eb5a621d8ecc38: Deleting consensus metadata
I20260926 14:54:39.447418 20456 catalog_manager.cc:4841] TS 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081): tablet 34104a3f89d44cd9a6261629cea906b2 (table test-table [id=52cc1f02ceec414685966f98c6963afe]) successfully deleted
I20260926 14:54:39.449396 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 20799
I20260926 14:54:39.453795 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 20437
I20260926 14:54:59.459126 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37971
--webserver_interface=127.5.15.190
--webserver_port=37471
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:59.536998 20960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:59.537189 20960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:59.537210 20960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:59.538410 20960 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:54:59.538450 20960 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:54:59.538463 20960 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:54:59.538475 20960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:54:59.538488 20960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:59.538501 20960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:59.538554 20960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:54:59.538571 20960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:54:59.540079 20960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:37971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:59.540335 20960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:59.540563 20960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:59.545544 20966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.545599 20968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.545569 20965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:59.545728 20960 server_base.cc:1034] running on GCE node
I20260926 14:54:59.545967 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:59.546244 20960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:59.547405 20960 hybrid_clock.cc:648] HybridClock initialized: now 1790434499547391 us; error 41 us; skew 500 ppm
Sep 26 14:54:59 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434499, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:59.555156 20960 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:54:59.555630 20960 webserver.cc:460] Webserver started at http://127.5.15.190:37471/ using document root <none> and password file <none>
I20260926 14:54:59.555852 20960 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:59.555920 20960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:59.557211 20960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:59.557845 20975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.558012 20960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.558125 20960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b45a4d9f2df04f908d7ff4ef7e2df789"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "08d8ada65ac91d5a1bfe4abf9fd8fbee"
server_key_iv: "ffd6476a8085928f3124954b88be712f"
server_key_version: "encryptionkey@0"
I20260926 14:54:59.558476 20960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:59.572866 20960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:59.574676 20960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:59.574868 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:59.578920 20960 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:37971
I20260926 14:54:59.578981 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:37971 every 8 connection(s)
I20260926 14:54:59.579365 20960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:54:59.580760 21028 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:54:59.581398 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap starting.
I20260926 14:54:59.584086 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 20960
I20260926 14:54:59.584355 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:41671
--local_ip_for_outbound_sockets=127.5.15.129
--tserver_master_addrs=127.5.15.190:37971
--webserver_port=34725
--webserver_interface=127.5.15.129
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:54:59.589828 21028 log.cc:824] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:59.592130 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:59.592422 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Bootstrap complete.
I20260926 14:54:59.594599 21028 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:59.594904 21028 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45a4d9f2df04f908d7ff4ef7e2df789, State: Initialized, Role: FOLLOWER
I20260926 14:54:59.595086 21028 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:59.595163 21028 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:54:59.595216 21028 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:54:59.595263 21028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:54:59.596203 21028 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:59.596323 21028 leader_election.cc:302] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45a4d9f2df04f908d7ff4ef7e2df789; no voters:
I20260926 14:54:59.596516 21028 leader_election.cc:288] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:54:59.596554 21032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:54:59.596665 21032 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [term 3 LEADER]: Becoming Leader. State: Replica: b45a4d9f2df04f908d7ff4ef7e2df789, State: Running, Role: LEADER
I20260926 14:54:59.596757 21032 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } }
I20260926 14:54:59.596787 21028 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:54:59.597021 21032 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45a4d9f2df04f908d7ff4ef7e2df789. Latest consensus state: current_term: 3 leader_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } } }
I20260926 14:54:59.597100 21032 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:59.597237 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45a4d9f2df04f908d7ff4ef7e2df789" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37971 } } }
I20260926 14:54:59.597313 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789 [sys.catalog]: This master's current role is: LEADER
I20260926 14:54:59.597581 21036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:54:59.598397 21036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:54:59.598557 21036 catalog_manager.cc:1259] Loaded cluster ID: 19e166f0bc62484dbc5a81ed3fe3293f
I20260926 14:54:59.598726 21036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:54:59.599910 21036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:54:59.600073 21036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Loaded TSK: 0
I20260926 14:54:59.600123 21036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b45a4d9f2df04f908d7ff4ef7e2df789: Loaded TSK: 1
F20260926 14:54:59.600358 21036 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f97af6b19ad google::LogMessage::Fail() at ??:0
@ 0x7f97af6b488a google::LogMessage::SendToLog() at ??:0
@ 0x7f97af6b14de google::LogMessage::Flush() at ??:0
@ 0x7f97af6b2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f97b1e1e104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f97b1e1bd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f97b760b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f97b760b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f97b760b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f97b75d1c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f97b75dbad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f97b157f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f97b1574954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f97b23d86db start_thread at ??:0
@ 0x7f97ae1af61f clone at ??:0
W20260926 14:54:59.682408 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:59.682592 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:59.682613 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:59.683753 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:59.683792 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:59.683818 21030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:54:59.685212 21030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=34725
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:59.685437 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:59.685643 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:59.690575 21072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.690629 21070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.690631 21069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:59.690970 21030 server_base.cc:1034] running on GCE node
I20260926 14:54:59.691200 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:59.691484 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:59.692662 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790434499692655 us; error 36 us; skew 500 ppm
Sep 26 14:54:59 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434499, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:59.695613 21030 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:54:59.696034 21030 webserver.cc:460] Webserver started at http://127.5.15.129:34725/ using document root <none> and password file <none>
I20260926 14:54:59.696255 21030 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:59.696323 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:59.697685 21030 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:59.698495 21079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.698670 21030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:54:59.698791 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "789644cf2b6440e3bd7883849cf515ad"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "b3f9069c23f053c0e1a969ee88bee328"
server_key_iv: "8a0811f0deadb8ba2e9c5743a3706b10"
server_key_version: "encryptionkey@0"
I20260926 14:54:59.699138 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:59.706919 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:59.708983 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:59.709136 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:59.709344 21030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:59.709785 21086 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:54:59.711597 21030 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:54:59.711655 21030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 14:54:59.711692 21030 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:54:59.712291 21030 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:54:59.712345 21030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 14:54:59.712417 21086 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap starting.
I20260926 14:54:59.714231 21086 log.cc:824] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Log is configured to *not* fsync() on all Append() calls
I20260926 14:54:59.717151 21086 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:54:59.717451 21086 tablet_bootstrap.cc:492] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Bootstrap complete.
I20260926 14:54:59.717602 21086 ts_tablet_manager.cc:1397] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 14:54:59.719481 21030 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:41671
I20260926 14:54:59.719569 21194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:41671 every 8 connection(s)
I20260926 14:54:59.719436 21086 raft_consensus.cc:348] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:59.719770 21086 raft_consensus.cc:729] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789644cf2b6440e3bd7883849cf515ad, State: Initialized, Role: FOLLOWER
I20260926 14:54:59.719868 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:54:59.719941 21086 consensus_queue.cc:260] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:54:59.720273 21086 ts_tablet_manager.cc:1428] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
I20260926 14:54:59.729406 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21030
I20260926 14:54:59.729660 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:46423
--local_ip_for_outbound_sockets=127.5.15.130
--tserver_master_addrs=127.5.15.190:37971
--webserver_port=35249
--webserver_interface=127.5.15.130
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:59.787459 21198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:54:59.720514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:35205)
0926 14:54:59.720653 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.720679 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 14:54:59.720861 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:54:59.787261 (+ 66400us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":92,"threads_started":1}
W20260926 14:54:59.787678 21195 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:37971 (0 consecutive failures): Network error: Failed to ping master at 127.5.15.190:37971: Client connection negotiation failed: client connection to 127.5.15.190:37971: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:54:59.788023 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.787903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:45959)
0926 14:54:59.787956 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.787996 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:54:59.788385 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.788305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:56327)
0926 14:54:59.788349 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.788371 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:54:59.788511 21195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:54:59.811724 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:59.811924 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:59.811962 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:59.813078 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:59.813138 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:59.813180 21199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:54:59.814725 21199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:46423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=35249
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:59.815066 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:59.815274 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:59.820060 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:59.820173 21199 server_base.cc:1034] running on GCE node
W20260926 14:54:59.820091 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.820057 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:59.820504 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:59.820751 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:59.821918 21199 hybrid_clock.cc:648] HybridClock initialized: now 1790434499821905 us; error 26 us; skew 500 ppm
Sep 26 14:54:59 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434499, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:59.824383 21199 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:54:59.824787 21199 webserver.cc:460] Webserver started at http://127.5.15.130:35249/ using document root <none> and password file <none>
I20260926 14:54:59.824965 21199 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:59.825003 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:59.826191 21199 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:54:59.826707 21214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.826879 21199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.826941 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "e2217900e9424a15865df67e28879b71"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "e1a9dc83a6eb02bdd7807a0cc0480bdf"
server_key_iv: "d533d8b88c6efaff7b468e64472be3e3"
server_key_version: "encryptionkey@0"
I20260926 14:54:59.827252 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:59.843472 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:59.845659 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:59.845758 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:59.845964 21199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:59.846261 21199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:59.846312 21199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.846351 21199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:59.846382 21199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.852083 21199 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:46423
I20260926 14:54:59.852135 21327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:46423 every 8 connection(s)
I20260926 14:54:59.852437 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:54:59.853375 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.853023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:48453)
0926 14:54:59.853176 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.853241 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
W20260926 14:54:59.853763 21328 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:37971 (0 consecutive failures): Network error: Failed to ping master at 127.5.15.190:37971: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
I20260926 14:54:59.854041 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.853950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:55883)
0926 14:54:59.853994 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.854022 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 14:54:59.854377 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.854303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:60617)
0926 14:54:59.854341 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.854363 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:54:59.854506 21328 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:54:59.854647 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21199
I20260926 14:54:59.854933 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:40081
--local_ip_for_outbound_sockets=127.5.15.131
--tserver_master_addrs=127.5.15.190:37971
--webserver_port=38891
--webserver_interface=127.5.15.131
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:34083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:54:59.932482 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:54:59.932663 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:54:59.932684 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:54:59.933816 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:54:59.933856 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:54:59.933882 21331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:54:59.935309 21331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:34083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:40081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=38891
--tserver_master_addrs=127.5.15.190:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:54:59.935488 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:54:59.935714 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:54:59.940584 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.940630 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:54:59.940624 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:54:59.940730 21331 server_base.cc:1034] running on GCE node
I20260926 14:54:59.941004 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:54:59.941262 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:54:59.942425 21331 hybrid_clock.cc:648] HybridClock initialized: now 1790434499942408 us; error 30 us; skew 500 ppm
Sep 26 14:54:59 dist-test-slave-s4p1 krb5kdc[19889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434499, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:54:59.944908 21331 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:54:59.945310 21331 webserver.cc:460] Webserver started at http://127.5.15.131:38891/ using document root <none> and password file <none>
I20260926 14:54:59.945526 21331 fs_manager.cc:359] Metadata directory not provided
I20260926 14:54:59.945593 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:54:59.946875 21331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:54:59.947450 21346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.947624 21331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:54:59.947718 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38"
format_stamp: "Formatted at 2026-09-26 14:54:18 on dist-test-slave-s4p1"
server_key: "be7805aa3891ac3a34a6d3ee817983a1"
server_key_iv: "e80b575dcb0d86297bf6152676c65942"
server_key_version: "encryptionkey@0"
I20260926 14:54:59.948037 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:54:59.959738 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:54:59.961905 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:54:59.962054 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:54:59.962263 21331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:54:59.962553 21331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:54:59.962608 21331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.962647 21331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:54:59.962673 21331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:54:59.968604 21331 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:40081
I20260926 14:54:59.968660 21459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:40081 every 8 connection(s)
I20260926 14:54:59.968961 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:54:59.969467 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21331
I20260926 14:54:59.969956 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.969537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:35607)
0926 14:54:59.969729 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.969794 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
W20260926 14:54:59.970403 21460 heartbeater.cc:644] Failed to heartbeat to 127.5.15.190:37971 (0 consecutive failures): Network error: Failed to ping master at 127.5.15.190:37971: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
I20260926 14:54:59.970710 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.970624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:57487)
0926 14:54:59.970671 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.970694 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:54:59.971050 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:54:59.970960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:54735)
0926 14:54:59.971006 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 14:54:59.971035 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:54:59.971182 21460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:55:00.789222 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:00.788931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:34491)
0926 14:55:00.789126 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:00.789161 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":56,"threads_started":1}
I20260926 14:55:00.855015 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:00.854807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:41381)
0926 14:55:00.854937 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:00.854969 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260926 14:55:00.971690 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:00.971505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:57761)
0926 14:55:00.971612 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:00.971644 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260926 14:55:01.148159 21467 raft_consensus.cc:482] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:01.148283 21467 raft_consensus.cc:504] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:55:01.148697 21467 leader_election.cc:288] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2217900e9424a15865df67e28879b71 (127.5.15.130:46423), 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081)
Sep 26 14:55:01 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434499, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.130@KRBTEST.COM
Sep 26 14:55:01 dist-test-slave-s4p1 krb5kdc[19889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434499, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.131@KRBTEST.COM
I20260926 14:55:01.159772 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.149169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:39133 (local address 127.5.15.131:40081)
0926 14:55:01.149357 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:01.149365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:01.149391 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:55:01.149827 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:01.149829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:01.149866 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:01.150028 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:01.151045 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.152027 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.153109 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.153457 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.153546 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:01.156344 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:01.156347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:01.156347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:01.156357 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:01.157987 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158483 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:01.158486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:01.158486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:01.158569 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158915 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:01.158916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:01.158916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:01.159143 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:01.159234 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:01.159541 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 14:55:01.159663 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":54,"threads_started":1}
I20260926 14:55:01.159770 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.148890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:46423 (local address 127.5.15.129:54375)
0926 14:55:01.148978 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:01.148993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:01.149082 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:01.149904 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:01.149907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:01.150388 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:01.150702 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:01.150709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.152171 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.152172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:01.152981 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:01.152983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.153634 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.153635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:01.153692 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:01.154182 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:01.154190 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:01.155647 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:01.158011 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:01.158020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:01.158368 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:01.158687 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:01.158688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:01.158814 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:01.159257 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:01.159258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:01.159374 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 14:55:01.159472 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:55:01.159658 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":88}
I20260926 14:55:01.159874 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.149040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:54375 (local address 127.5.15.130:46423)
0926 14:55:01.149191 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:01.149210 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:01.149238 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:55:01.149695 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:01.149697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:01.149735 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:01.149880 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:01.150871 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.152081 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.153085 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.153513 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.153651 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:01.155759 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:01.155764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:01.155764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:01.155781 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:01.157896 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158469 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:01.158473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:01.158473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:01.158583 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158898 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:01.158899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:01.158900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:01.159151 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:01.159255 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:01.159554 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 14:55:01.159735 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260926 14:55:01.160249 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.149029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:40081 (local address 127.5.15.129:39133)
0926 14:55:01.149157 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:01.149168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:01.149230 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:01.150013 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:01.150013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:01.150382 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:01.150937 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:01.150940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.152141 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.152143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:01.152982 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:01.152985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:01.153560 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:01.153561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:01.153691 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:01.154351 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:01.154358 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:01.156124 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:01.158068 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:01.158069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:01.158369 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:01.158639 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:01.158640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:01.158641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:01.158821 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:01.159220 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:01.159226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:01.159374 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 14:55:01.159472 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:55:01.159744 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":161,"thread_start_us":55,"threads_started":1}
W20260926 14:55:01.160708 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2217900e9424a15865df67e28879b71 (127.5.15.130:46423): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
W20260926 14:55:01.161013 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
I20260926 14:55:01.161054 21082 leader_election.cc:302] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789644cf2b6440e3bd7883849cf515ad; no voters: 21eb5a83aadb4a07b9eb5a621d8ecc38, e2217900e9424a15865df67e28879b71
I20260926 14:55:01.161154 21467 raft_consensus.cc:2726] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:55:01.789935 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.789734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:55305)
0926 14:55:01.789857 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:01.789890 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260926 14:55:01.855638 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.855444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:38307)
0926 14:55:01.855572 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:01.855604 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260926 14:55:01.972295 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:01.972075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:53291)
0926 14:55:01.972229 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:01.972261 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260926 14:55:02.687072 21474 raft_consensus.cc:482] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:02.687162 21474 raft_consensus.cc:504] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:55:02.687280 21474 leader_election.cc:288] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2217900e9424a15865df67e28879b71 (127.5.15.130:46423), 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081)
W20260926 14:55:02.687680 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
W20260926 14:55:02.687769 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2217900e9424a15865df67e28879b71 (127.5.15.130:46423): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
I20260926 14:55:02.687803 21082 leader_election.cc:302] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789644cf2b6440e3bd7883849cf515ad; no voters: 21eb5a83aadb4a07b9eb5a621d8ecc38, e2217900e9424a15865df67e28879b71
I20260926 14:55:02.687903 21474 raft_consensus.cc:2726] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:55:02.790491 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:02.790303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:33295)
0926 14:55:02.790411 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:02.790442 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260926 14:55:02.856179 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:02.856008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:58357)
0926 14:55:02.856115 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:02.856143 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260926 14:55:02.972967 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:02.972771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:40779)
0926 14:55:02.972900 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:02.972931 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260926 14:55:03.791199 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:03.790988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:58863)
0926 14:55:03.791127 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:03.791157 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260926 14:55:03.856778 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:03.856592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:37095)
0926 14:55:03.856710 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:03.856741 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260926 14:55:03.973693 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:03.973464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:37033)
0926 14:55:03.973607 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:03.973650 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260926 14:55:04.791944 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:04.791638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:59083)
0926 14:55:04.791862 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:04.791897 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":62,"threads_started":1}
I20260926 14:55:04.857419 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:04.857215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:59299)
0926 14:55:04.857342 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:04.857377 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260926 14:55:04.974440 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:04.974183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:42885)
0926 14:55:04.974376 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:04.974407 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":66,"threads_started":1}
I20260926 14:55:05.792858 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:05.792551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:39457)
0926 14:55:05.792763 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:05.792798 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":59,"threads_started":1}
I20260926 14:55:05.858147 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:05.857899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:47441)
0926 14:55:05.858032 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:05.858070 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260926 14:55:05.889151 21486 raft_consensus.cc:482] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:05.889247 21486 raft_consensus.cc:504] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:55:05.889366 21486 leader_election.cc:288] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2217900e9424a15865df67e28879b71 (127.5.15.130:46423), 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081)
W20260926 14:55:05.889776 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
W20260926 14:55:05.889845 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2217900e9424a15865df67e28879b71 (127.5.15.130:46423): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
I20260926 14:55:05.889863 21082 leader_election.cc:302] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789644cf2b6440e3bd7883849cf515ad; no voters: 21eb5a83aadb4a07b9eb5a621d8ecc38, e2217900e9424a15865df67e28879b71
I20260926 14:55:05.889928 21486 raft_consensus.cc:2726] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:55:05.975299 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:05.975062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:35939)
0926 14:55:05.975224 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:05.975261 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260926 14:55:06.793481 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:06.793282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:36001)
0926 14:55:06.793406 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:06.793437 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 14:55:06.858732 21489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:06.858550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:33141)
0926 14:55:06.858661 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:06.858694 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260926 14:55:06.975906 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:06.975713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:57853)
0926 14:55:06.975822 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:06.975872 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260926 14:55:07.794201 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:07.794014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:47741)
0926 14:55:07.794129 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:07.794158 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260926 14:55:07.859359 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:07.859133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:45907)
0926 14:55:07.859252 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:07.859283 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260926 14:55:07.976511 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:07.976330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:50943)
0926 14:55:07.976442 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:07.976475 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260926 14:55:08.794858 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:08.794604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:53971)
0926 14:55:08.794722 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:08.794804 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260926 14:55:08.860041 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:08.859827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:50261)
0926 14:55:08.859971 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:08.860004 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260926 14:55:08.977144 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:08.976961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:48961)
0926 14:55:08.977076 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:08.977106 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260926 14:55:09.795805 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:09.795457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:33515)
0926 14:55:09.795707 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:09.795757 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":63,"threads_started":1}
I20260926 14:55:09.860841 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:09.860615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:36395)
0926 14:55:09.860760 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:09.860802 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260926 14:55:09.977839 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:09.977627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:45767)
0926 14:55:09.977756 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:09.977799 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260926 14:55:10.647882 21500 raft_consensus.cc:482] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:10.647976 21500 raft_consensus.cc:504] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2217900e9424a15865df67e28879b71" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 46423 } } peers { permanent_uuid: "21eb5a83aadb4a07b9eb5a621d8ecc38" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 40081 } } peers { permanent_uuid: "789644cf2b6440e3bd7883849cf515ad" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 41671 } }
I20260926 14:55:10.648169 21500 leader_election.cc:288] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2217900e9424a15865df67e28879b71 (127.5.15.130:46423), 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081)
W20260926 14:55:10.648559 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21eb5a83aadb4a07b9eb5a621d8ecc38 (127.5.15.131:40081): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
W20260926 14:55:10.648635 21082 leader_election.cc:341] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2217900e9424a15865df67e28879b71 (127.5.15.130:46423): Not found: Tablet not found: 34104a3f89d44cd9a6261629cea906b2
I20260926 14:55:10.648651 21082 leader_election.cc:302] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 789644cf2b6440e3bd7883849cf515ad; no voters: 21eb5a83aadb4a07b9eb5a621d8ecc38, e2217900e9424a15865df67e28879b71
I20260926 14:55:10.648720 21500 raft_consensus.cc:2726] T 34104a3f89d44cd9a6261629cea906b2 P 789644cf2b6440e3bd7883849cf515ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:55:10.796592 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:10.796375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:44251)
0926 14:55:10.796508 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:10.796543 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260926 14:55:10.861509 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:10.861263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:51767)
0926 14:55:10.861395 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:10.861432 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260926 14:55:10.978703 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:10.978440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:40627)
0926 14:55:10.978584 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:10.978621 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260926 14:55:11.797463 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:11.797236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:55209)
0926 14:55:11.797383 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:11.797417 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260926 14:55:11.862123 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:11.861929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:56243)
0926 14:55:11.862057 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:11.862088 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 14:55:11.979441 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:11.979217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:48461)
0926 14:55:11.979355 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:11.979388 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":55,"threads_started":1}
I20260926 14:55:12.798290 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:12.798026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:40147)
0926 14:55:12.798209 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:12.798243 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":112,"threads_started":1}
I20260926 14:55:12.862890 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:12.862667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:56407)
0926 14:55:12.862811 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:12.862846 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":115,"threads_started":1}
I20260926 14:55:12.980065 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:12.979867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:45269)
0926 14:55:12.979991 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:12.980024 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260926 14:55:13.798997 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:13.798739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:43837)
0926 14:55:13.798915 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:13.798950 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":79,"threads_started":1}
I20260926 14:55:13.863518 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:13.863314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:36197)
0926 14:55:13.863443 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:13.863475 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260926 14:55:13.980827 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:13.980572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.131:40735)
0926 14:55:13.980730 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:13.980764 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260926 14:55:14.799671 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:14.799451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.129:43547)
0926 14:55:14.799594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:14.799627 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260926 14:55:14.864202 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:14.863943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:37971 (local address 127.5.15.130:56915)
0926 14:55:14.864123 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:14.864155 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.190:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260926 14:55:14.971683 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21030
I20260926 14:55:14.976264 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21199
I20260926 14:55:14.980859 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21331
2026-09-26T14:55:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58186 ms)
[----------] 30 tests from SecurityITest (313619 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:15 dist-test-slave-s4p1 krb5kdc[21517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:15 dist-test-slave-s4p1 krb5kdc[21517](info): set up 2 sockets
Sep 26 14:55:15 dist-test-slave-s4p1 krb5kdc[21517](info): commencing operation
krb5kdc: starting...
W20260926 14:55:16.002662 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:16.117457 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:32809
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:32809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:16.196622 21530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:16.196815 21530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:16.196835 21530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:16.197937 21530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:16.197976 21530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:16.197990 21530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:16.198004 21530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:16.198072 21530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:16.199528 21530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:32809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:32809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:16.199770 21530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:16.199999 21530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:16.204972 21538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.205011 21536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:16.205179 21530 server_base.cc:1034] running on GCE node
W20260926 14:55:16.204967 21535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:16.205474 21530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:16.205708 21530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:16.206820 21530 hybrid_clock.cc:648] HybridClock initialized: now 1790434516206805 us; error 27 us; skew 500 ppm
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:16.209074 21530 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:16.209529 21530 webserver.cc:460] Webserver started at http://127.5.15.190:46341/ using document root <none> and password file <none>
I20260926 14:55:16.209765 21530 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:16.209834 21530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:16.209981 21530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:16.211047 21530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "8481001a22664d008f54138c1d2eb651"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "e9d29a9a672a338cd43b98e8544a1380"
server_key_iv: "5de7631ef4c0907f138bceb0daf561fa"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.211422 21530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "8481001a22664d008f54138c1d2eb651"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "e9d29a9a672a338cd43b98e8544a1380"
server_key_iv: "5de7631ef4c0907f138bceb0daf561fa"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.212798 21530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:16.213443 21545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.213600 21530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:16.213701 21530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "8481001a22664d008f54138c1d2eb651"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "e9d29a9a672a338cd43b98e8544a1380"
server_key_iv: "5de7631ef4c0907f138bceb0daf561fa"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.213770 21530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:16.236037 21530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:16.237938 21530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:16.238097 21530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:16.242357 21530 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:32809
I20260926 14:55:16.242432 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:32809 every 8 connection(s)
I20260926 14:55:16.242893 21530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:16.243742 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:16.245814 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651: Bootstrap starting.
I20260926 14:55:16.246325 21598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:16.246592 21598 log.cc:824] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:16.247239 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651: No bootstrap required, opened a new log
I20260926 14:55:16.248740 21598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481001a22664d008f54138c1d2eb651" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32809 } }
I20260926 14:55:16.248879 21598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:16.248903 21598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8481001a22664d008f54138c1d2eb651, State: Initialized, Role: FOLLOWER
I20260926 14:55:16.249033 21598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481001a22664d008f54138c1d2eb651" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32809 } }
I20260926 14:55:16.249115 21598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:16.249137 21598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:16.249166 21598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:16.249632 21598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481001a22664d008f54138c1d2eb651" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32809 } }
I20260926 14:55:16.249713 21598 leader_election.cc:302] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8481001a22664d008f54138c1d2eb651; no voters:
I20260926 14:55:16.249830 21598 leader_election.cc:288] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:16.249909 21601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:16.250087 21598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:16.250073 21601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [term 1 LEADER]: Becoming Leader. State: Replica: 8481001a22664d008f54138c1d2eb651, State: Running, Role: LEADER
I20260926 14:55:16.250201 21601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481001a22664d008f54138c1d2eb651" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32809 } }
I20260926 14:55:16.250574 21603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8481001a22664d008f54138c1d2eb651. Latest consensus state: current_term: 1 leader_uuid: "8481001a22664d008f54138c1d2eb651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481001a22664d008f54138c1d2eb651" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32809 } } }
I20260926 14:55:16.250679 21603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:16.250700 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8481001a22664d008f54138c1d2eb651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8481001a22664d008f54138c1d2eb651" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 32809 } } }
I20260926 14:55:16.250833 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:16.251235 21610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:16.251771 21610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:16.252166 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21530
I20260926 14:55:16.252252 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:16.252446 5182 external_mini_cluster.cc:1428] Setting key c3f8b0b04d0019a6fe11b2c27e6039aa
I20260926 14:55:16.255985 21610 catalog_manager.cc:1347] Generated new cluster ID: 12cbc5236d4a458b8c906c2c3036e90e
I20260926 14:55:16.256042 21610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:16.261139 21610 catalog_manager.cc:1370] Generated new certificate authority record
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:16.261823 21610 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:16.265806 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.253087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.5.15.190:32809)
0926 14:55:16.253237 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.253246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.254983 (+ 1737us) server_negotiation.cc:407] Connection header received
0926 14:55:16.255105 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.255110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.255150 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.255303 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:16.255953 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.257406 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.258558 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.259157 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.259260 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.262092 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:16.262096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:16.262097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:16.262109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:16.264088 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.264466 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.264469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.264469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.264577 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.264779 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.264780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.264780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.265071 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:16.265202 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.265323 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.265574 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":99,"thread_start_us":50,"threads_started":1}
I20260926 14:55:16.268977 21610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651: Generated new TSK 0
I20260926 14:55:16.269146 21610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:55:16.298354 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32809
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:16.382833 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:16.383092 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:16.383119 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:16.384284 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:16.384325 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:16.384351 21624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:55:16.385794 21624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:16.386054 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:16.386299 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:16.391111 21629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.391139 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.391129 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:16.391456 21624 server_base.cc:1034] running on GCE node
I20260926 14:55:16.391660 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:16.391918 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:16.393101 21624 hybrid_clock.cc:648] HybridClock initialized: now 1790434516393093 us; error 35 us; skew 500 ppm
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:16.395622 21624 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:55:16.396040 21624 webserver.cc:460] Webserver started at http://127.5.15.129:37391/ using document root <none> and password file <none>
I20260926 14:55:16.396231 21624 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:16.396270 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:16.396374 21624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:16.397351 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "c0f3924b9954004dc465ecf622606290"
server_key_iv: "826c3fb7b8b795c36bd7d4d27c6b87a3"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.397641 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "c0f3924b9954004dc465ecf622606290"
server_key_iv: "826c3fb7b8b795c36bd7d4d27c6b87a3"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.398902 21624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:16.399585 21639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.399765 21624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.399830 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "c0f3924b9954004dc465ecf622606290"
server_key_iv: "826c3fb7b8b795c36bd7d4d27c6b87a3"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.399879 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:16.418036 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:16.420365 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:16.420466 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:16.420617 21624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:16.420907 21624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:16.420936 21624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.420956 21624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:16.421000 21624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.426532 21624 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:36747
I20260926 14:55:16.426579 21752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:36747 every 8 connection(s)
I20260926 14:55:16.426864 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:16.433729 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21624
I20260926 14:55:16.433851 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:55:16.434041 5182 external_mini_cluster.cc:1428] Setting key ead9b861b37e2a67ee4fc6dc084a48ba
I20260926 14:55:16.436009 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.427516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:49793 (local address 127.5.15.190:32809)
0926 14:55:16.427577 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.427579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.427751 (+ 172us) server_negotiation.cc:407] Connection header received
0926 14:55:16.428191 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.428192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.428201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.428259 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:16.429117 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.429616 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.430638 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.430901 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.430951 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.433039 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:16.433040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:16.433040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:16.433045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:16.434536 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.435051 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.435052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.435052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.435108 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.435397 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.435398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.435398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.435497 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:16.435548 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.435861 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.435935 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:55:16.436301 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.427487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:32809 (local address 127.5.15.129:49793)
0926 14:55:16.427628 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.427666 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.428113 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.428307 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.428310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.428856 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:16.429039 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.429043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.429695 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.429695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.430545 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.430548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.430960 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.430961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.431063 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.431584 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:16.431592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:16.432961 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:16.434614 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:16.434620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:16.434622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:16.434981 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:16.435177 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:16.435178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:16.435179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:16.435334 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:16.435559 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:16.435565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:16.435728 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.435837 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.436022 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260926 14:55:16.436978 21753 heartbeater.cc:347] Connected to a master server at 127.5.15.190:32809
I20260926 14:55:16.437103 21753 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:16.437296 21753 heartbeater.cc:511] Master 127.5.15.190:32809 requested a full tablet report, sending...
I20260926 14:55:16.437810 21562 ts_manager.cc:194] Registered new tserver with Master: c2fd1fbbad9b4634a318b2f5f3f1984c (127.5.15.129:36747)
I20260926 14:55:16.438428 21562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:49793
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:55:16.459641 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32809
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:16.537086 21758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:16.537302 21758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:16.537336 21758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:16.538491 21758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:16.538556 21758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:16.538599 21758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:55:16.539991 21758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:16.540231 21758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:16.540444 21758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:16.545353 21766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.545409 21764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.545353 21763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:16.545553 21758 server_base.cc:1034] running on GCE node
I20260926 14:55:16.545786 21758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:16.546048 21758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:16.547204 21758 hybrid_clock.cc:648] HybridClock initialized: now 1790434516547185 us; error 29 us; skew 500 ppm
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:16.549456 21758 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:55:16.549950 21758 webserver.cc:460] Webserver started at http://127.5.15.130:46191/ using document root <none> and password file <none>
I20260926 14:55:16.550163 21758 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:16.550225 21758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:16.550370 21758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:16.551573 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "3b7f3d7531ea5587fdb20f9a3671eb90"
server_key_iv: "73165bc20f76333201ae417403fb893a"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.551956 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "3b7f3d7531ea5587fdb20f9a3671eb90"
server_key_iv: "73165bc20f76333201ae417403fb893a"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.553273 21758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:55:16.553992 21773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.554179 21758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:16.554273 21758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "3b7f3d7531ea5587fdb20f9a3671eb90"
server_key_iv: "73165bc20f76333201ae417403fb893a"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.554354 21758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:16.568818 21758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:16.570555 21758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:16.570712 21758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:16.570994 21758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:16.571290 21758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:16.571342 21758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.571373 21758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:16.571400 21758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.577307 21758 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:40733
I20260926 14:55:16.577440 21886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:40733 every 8 connection(s)
I20260926 14:55:16.577677 21758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:16.584415 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21758
I20260926 14:55:16.584518 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:55:16.584743 5182 external_mini_cluster.cc:1428] Setting key 1155175f1bc07fadd79825b01c5bc1ba
I20260926 14:55:16.586787 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.578315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:35339 (local address 127.5.15.190:32809)
0926 14:55:16.578392 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.578395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.578628 (+ 233us) server_negotiation.cc:407] Connection header received
0926 14:55:16.579180 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.579181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.579190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.579241 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:16.580092 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.580584 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.581510 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.581693 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.581747 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.583833 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:16.583835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:16.583835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:16.583841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:16.585443 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.585818 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.585819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.585819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.585883 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.586103 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.586104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.586104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.586242 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:16.586312 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.586532 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.586623 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:55:16.586791 21889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.578294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:32809 (local address 127.5.15.130:35339)
0926 14:55:16.578471 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.578529 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.579089 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.579294 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.579297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.579836 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:16.580016 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.580020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.580660 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.580661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.581429 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.581431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.581765 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.581765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.581861 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.582382 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:16.582397 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:16.583752 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:16.585530 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:16.585536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:16.585537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:16.585744 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:16.585951 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:16.585951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:16.585952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:16.586042 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:16.586305 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:16.586309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:16.586411 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.586488 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.586597 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260926 14:55:16.587441 21887 heartbeater.cc:347] Connected to a master server at 127.5.15.190:32809
I20260926 14:55:16.587539 21887 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:16.587719 21887 heartbeater.cc:511] Master 127.5.15.190:32809 requested a full tablet report, sending...
I20260926 14:55:16.588099 21562 ts_manager.cc:194] Registered new tserver with Master: bb0c4d2e07d34e8fb08c10bed9ddcdbc (127.5.15.130:40733)
I20260926 14:55:16.588526 21562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:35339
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:55:16.609726 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32809
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:16.687494 21892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:16.687712 21892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:16.687750 21892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:16.688846 21892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:16.688910 21892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:16.688953 21892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:55:16.690328 21892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:32809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:16.690565 21892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:16.690804 21892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:16.695354 21897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.695293 21898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.695291 21900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:16.695433 21892 server_base.cc:1034] running on GCE node
I20260926 14:55:16.695694 21892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:16.695919 21892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:16.697069 21892 hybrid_clock.cc:648] HybridClock initialized: now 1790434516697060 us; error 41 us; skew 500 ppm
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:16.699409 21892 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:55:16.699847 21892 webserver.cc:460] Webserver started at http://127.5.15.131:46623/ using document root <none> and password file <none>
I20260926 14:55:16.700067 21892 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:16.700136 21892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:16.700289 21892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:16.701269 21892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "95e2ba1f02394f1cb24f903cfce819f2"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "c0eefda83b3096080e6aa6668dc587fe"
server_key_iv: "0e0c2720c864bd61b8ea73c57cc9cb14"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.701619 21892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "95e2ba1f02394f1cb24f903cfce819f2"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "c0eefda83b3096080e6aa6668dc587fe"
server_key_iv: "0e0c2720c864bd61b8ea73c57cc9cb14"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.703004 21892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:16.703692 21907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.703873 21892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:16.703936 21892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "95e2ba1f02394f1cb24f903cfce819f2"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "c0eefda83b3096080e6aa6668dc587fe"
server_key_iv: "0e0c2720c864bd61b8ea73c57cc9cb14"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.704020 21892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:16.712494 21892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:16.714519 21892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:16.714663 21892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:16.714885 21892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:16.715195 21892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:16.715226 21892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.715276 21892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:16.715304 21892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.720924 21892 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:35289
I20260926 14:55:16.720960 22020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:35289 every 8 connection(s)
I20260926 14:55:16.721261 21892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:55:16.723912 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 21892
I20260926 14:55:16.724015 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:55:16.724197 5182 external_mini_cluster.cc:1428] Setting key eac4d782111abc2224408c4ca7efadd4
Sep 26 14:55:16 dist-test-slave-s4p1 krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434516, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:16.730314 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.721948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:52547 (local address 127.5.15.190:32809)
0926 14:55:16.721990 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.721992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.722173 (+ 181us) server_negotiation.cc:407] Connection header received
0926 14:55:16.722635 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.722636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.722649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.722702 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:16.723620 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.724435 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.725602 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.725795 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.725850 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.727915 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:16.727917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:16.727917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:16.727924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:16.729282 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.729612 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.729612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.729613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.729658 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.729857 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.729857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.729857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.729950 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:16.729996 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.730200 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.730253 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:55:16.730523 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.721898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:32809 (local address 127.5.15.131:52547)
0926 14:55:16.722034 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.722078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.722526 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.722786 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.722789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.723328 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:16.723499 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.723504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.724546 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.724547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.725527 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.725530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.725870 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.725871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.725962 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.726432 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:16.726456 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:16.727832 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:16.729348 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:16.729353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:16.729354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:16.729551 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:16.729715 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:16.729716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:16.729716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:16.729804 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:16.730002 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:16.730005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:16.730103 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.730185 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.730291 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 14:55:16.731089 22021 heartbeater.cc:347] Connected to a master server at 127.5.15.190:32809
I20260926 14:55:16.731171 22021 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:16.731297 22021 heartbeater.cc:511] Master 127.5.15.190:32809 requested a full tablet report, sending...
I20260926 14:55:16.731673 21562 ts_manager.cc:194] Registered new tserver with Master: 95e2ba1f02394f1cb24f903cfce819f2 (127.5.15.131:35289)
I20260926 14:55:16.732033 21562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:52547
I20260926 14:55:16.735806 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:55:16.742632 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.737089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47576 (local address 127.5.15.190:32809)
0926 14:55:16.737135 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.737136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.737276 (+ 140us) server_negotiation.cc:407] Connection header received
0926 14:55:16.737334 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.737335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.737343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.737380 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:16.737892 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.738342 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.739171 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.739347 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.739395 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.740388 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:16.740389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:16.740389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:16.740394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:16.741683 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.742020 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.742021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.742021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.742065 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:16.742248 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:16.742248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:16.742249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:16.742366 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:16.742437 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.742528 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.742578 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:55:16.743986 21562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:55:16.745497 21562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:55:16.751926 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.748004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.5.15.130:40733)
0926 14:55:16.748155 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.748160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.748169 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:16.748208 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.748210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.748224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.748461 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:16.748533 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.749622 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.750928 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.751528 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.751584 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.751615 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.751810 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.751852 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260926 14:55:16.751929 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.747596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.5.15.131:35289)
0926 14:55:16.747758 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.747763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.747889 (+ 126us) server_negotiation.cc:407] Connection header received
0926 14:55:16.747973 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.747975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.747989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.748085 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:16.748239 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.748968 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.750094 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.750810 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.750885 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.750939 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.751833 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.751866 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260926 14:55:16.751914 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.747568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:40733 (local address 127.0.0.1:35804)
0926 14:55:16.747960 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.747969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.748006 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.748329 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.748330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.748332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:16.748386 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.748387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.749774 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.749776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.750843 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.750845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.751604 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.751605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.751696 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.751711 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.751777 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.751832 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"thread_start_us":41,"threads_started":1}
I20260926 14:55:16.752161 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.747568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36747 (local address 127.0.0.1:41908)
0926 14:55:16.747780 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.747816 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.747877 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.748390 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.748390 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.748392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:16.748430 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.748432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.749748 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.749749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.750808 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.750812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.751972 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.751972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.752029 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.752033 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.752065 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.752098 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260926 14:55:16.751914 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.747539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:35289 (local address 127.0.0.1:49176)
0926 14:55:16.747777 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.747812 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.747901 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.748095 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.748098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.748100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:16.748156 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.748158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.749077 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.749078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.749990 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.749992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.751701 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.751702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.751759 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.751763 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.751812 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.751846 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260926 14:55:16.752674 22033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.747795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.5.15.129:36747)
0926 14:55:16.748233 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.748236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.748247 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:55:16.748282 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.748284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.748299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.748509 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:16.748581 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.749631 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.750878 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.751601 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.752320 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.752539 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.752564 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.752603 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":90,"threads_started":1}
I20260926 14:55:16.752864 21955 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8a3c7fc51541139bdbbaab958c767e (DEFAULT_TABLE table=test-table [id=c901734fed24492e8f9613112038af65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:16.753003 21821 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8a3c7fc51541139bdbbaab958c767e (DEFAULT_TABLE table=test-table [id=c901734fed24492e8f9613112038af65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:16.753193 21955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8a3c7fc51541139bdbbaab958c767e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:16.753345 21821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8a3c7fc51541139bdbbaab958c767e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:16.753269 21687 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8a3c7fc51541139bdbbaab958c767e (DEFAULT_TABLE table=test-table [id=c901734fed24492e8f9613112038af65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:16.753530 21687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8a3c7fc51541139bdbbaab958c767e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:16.756001 22035 tablet_bootstrap.cc:492] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: Bootstrap starting.
I20260926 14:55:16.756438 22036 tablet_bootstrap.cc:492] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Bootstrap starting.
I20260926 14:55:16.756667 22037 tablet_bootstrap.cc:492] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c: Bootstrap starting.
I20260926 14:55:16.756814 22035 tablet_bootstrap.cc:654] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:16.757078 22036 tablet_bootstrap.cc:654] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:16.757300 22037 tablet_bootstrap.cc:654] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:16.757453 22035 log.cc:824] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:16.757715 22036 log.cc:824] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:16.757920 22037 log.cc:824] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:16.758479 22036 tablet_bootstrap.cc:492] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: No bootstrap required, opened a new log
I20260926 14:55:16.758562 22036 ts_tablet_manager.cc:1397] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:16.758621 22037 tablet_bootstrap.cc:492] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c: No bootstrap required, opened a new log
I20260926 14:55:16.758678 22037 ts_tablet_manager.cc:1397] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:16.759995 22037 raft_consensus.cc:348] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.759989 22036 raft_consensus.cc:348] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.760121 22037 raft_consensus.cc:374] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:16.760147 22037 raft_consensus.cc:729] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2fd1fbbad9b4634a318b2f5f3f1984c, State: Initialized, Role: FOLLOWER
I20260926 14:55:16.760152 22036 raft_consensus.cc:374] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:16.760192 22036 raft_consensus.cc:729] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e2ba1f02394f1cb24f903cfce819f2, State: Initialized, Role: FOLLOWER
I20260926 14:55:16.760299 22036 consensus_queue.cc:260] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.760295 22037 consensus_queue.cc:260] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.760581 22036 ts_tablet_manager.cc:1428] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:16.760592 22021 heartbeater.cc:503] Master 127.5.15.190:32809 was elected leader, sending a full tablet report...
I20260926 14:55:16.760684 22037 ts_tablet_manager.cc:1428] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:55:16.760700 21753 heartbeater.cc:503] Master 127.5.15.190:32809 was elected leader, sending a full tablet report...
I20260926 14:55:16.761006 22035 tablet_bootstrap.cc:492] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: No bootstrap required, opened a new log
I20260926 14:55:16.761399 22035 ts_tablet_manager.cc:1397] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 14:55:16.763114 22035 raft_consensus.cc:348] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.763279 22035 raft_consensus.cc:374] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:16.763322 22035 raft_consensus.cc:729] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0c4d2e07d34e8fb08c10bed9ddcdbc, State: Initialized, Role: FOLLOWER
I20260926 14:55:16.763422 22035 consensus_queue.cc:260] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.763661 22035 ts_tablet_manager.cc:1428] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:16.763710 21887 heartbeater.cc:503] Master 127.5.15.190:32809 was elected leader, sending a full tablet report...
I20260926 14:55:16.820747 22041 raft_consensus.cc:482] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:16.820849 22041 raft_consensus.cc:504] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.821149 22041 leader_election.cc:288] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2fd1fbbad9b4634a318b2f5f3f1984c (127.5.15.129:36747), bb0c4d2e07d34e8fb08c10bed9ddcdbc (127.5.15.130:40733)
I20260926 14:55:16.824049 22033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.821404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38081 (local address 127.5.15.129:36747)
0926 14:55:16.821463 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.821465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.821473 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:55:16.821500 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.821501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.821513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.821582 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:16.821764 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.822318 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.823262 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.823672 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.823733 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.823739 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.823938 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.823981 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:55:16.824074 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.821415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:34199 (local address 127.5.15.130:40733)
0926 14:55:16.821458 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.821460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.821611 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:55:16.821644 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.821645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.821658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.821717 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:16.821886 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.822394 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.823340 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.823753 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.823824 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.823831 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.823975 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.824014 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:55:16.824069 22023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.821324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36747 (local address 127.5.15.131:38081)
0926 14:55:16.821381 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.821391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.821431 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.821611 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.821612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.821615 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:16.821689 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.821691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.822405 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.822406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.823176 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.823178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.823756 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.823757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.823843 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.823860 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.823920 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.823979 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":24}
I20260926 14:55:16.824076 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.821324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:40733 (local address 127.5.15.131:34199)
0926 14:55:16.821534 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.821544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:16.821600 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:16.821749 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:16.821750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:16.821751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:16.821819 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.821820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.822470 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.822471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.823257 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:16.823260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.823828 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.823828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:16.823881 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.823884 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:16.823957 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:55:16.823991 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":20,"thread_start_us":114,"threads_started":1}
I20260926 14:55:16.824409 21707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8a3c7fc51541139bdbbaab958c767e" candidate_uuid: "95e2ba1f02394f1cb24f903cfce819f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" is_pre_election: true
I20260926 14:55:16.824571 21707 raft_consensus.cc:2454] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e2ba1f02394f1cb24f903cfce819f2 in term 0.
I20260926 14:55:16.824538 21841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8a3c7fc51541139bdbbaab958c767e" candidate_uuid: "95e2ba1f02394f1cb24f903cfce819f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" is_pre_election: true
I20260926 14:55:16.824710 21841 raft_consensus.cc:2454] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95e2ba1f02394f1cb24f903cfce819f2 in term 0.
I20260926 14:55:16.824831 21909 leader_election.cc:302] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e2ba1f02394f1cb24f903cfce819f2, c2fd1fbbad9b4634a318b2f5f3f1984c; no voters:
I20260926 14:55:16.824995 22041 raft_consensus.cc:2781] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:55:16.825043 22041 raft_consensus.cc:482] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:55:16.825060 22041 raft_consensus.cc:3037] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:16.825706 22041 raft_consensus.cc:504] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
I20260926 14:55:16.825840 22041 leader_election.cc:288] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [CANDIDATE]: Term 1 election: Requested vote from peers c2fd1fbbad9b4634a318b2f5f3f1984c (127.5.15.129:36747), bb0c4d2e07d34e8fb08c10bed9ddcdbc (127.5.15.130:40733)
I20260926 14:55:16.825999 21707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8a3c7fc51541139bdbbaab958c767e" candidate_uuid: "95e2ba1f02394f1cb24f903cfce819f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c"
I20260926 14:55:16.825999 21841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8a3c7fc51541139bdbbaab958c767e" candidate_uuid: "95e2ba1f02394f1cb24f903cfce819f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc"
I20260926 14:55:16.826072 21707 raft_consensus.cc:3037] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:16.826072 21841 raft_consensus.cc:3037] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:16.826975 21841 raft_consensus.cc:2454] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e2ba1f02394f1cb24f903cfce819f2 in term 1.
I20260926 14:55:16.826975 21707 raft_consensus.cc:2454] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e2ba1f02394f1cb24f903cfce819f2 in term 1.
I20260926 14:55:16.827136 21909 leader_election.cc:302] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95e2ba1f02394f1cb24f903cfce819f2, c2fd1fbbad9b4634a318b2f5f3f1984c; no voters:
I20260926 14:55:16.827243 22041 raft_consensus.cc:2781] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:16.827366 22041 raft_consensus.cc:686] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 1 LEADER]: Becoming Leader. State: Replica: 95e2ba1f02394f1cb24f903cfce819f2, State: Running, Role: LEADER
I20260926 14:55:16.827454 22041 consensus_queue.cc:237] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } }
W20260926 14:55:16.828200 21888 tablet.cc:2369] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:55:16.828194 21561 catalog_manager.cc:5495] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95e2ba1f02394f1cb24f903cfce819f2 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "95e2ba1f02394f1cb24f903cfce819f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2ba1f02394f1cb24f903cfce819f2" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 35289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 } health_report { overall_health: UNKNOWN } } }
I20260926 14:55:16.842099 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.840140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.5.15.131:35289)
0926 14:55:16.840192 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:16.840194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:16.840202 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:55:16.840234 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:16.840235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:16.840247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:16.840312 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:16.840429 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.840870 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.841515 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:16.841674 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:16.841724 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:16.841847 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:16.841925 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:16.841980 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:16.842009 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 14:55:16.842024 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:55:16.843634 21841 raft_consensus.cc:1261] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 1 FOLLOWER]: Refusing update from remote peer 95e2ba1f02394f1cb24f903cfce819f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:16.843636 21707 raft_consensus.cc:1261] T 5f8a3c7fc51541139bdbbaab958c767e P c2fd1fbbad9b4634a318b2f5f3f1984c [term 1 FOLLOWER]: Refusing update from remote peer 95e2ba1f02394f1cb24f903cfce819f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:16.843945 22045 consensus_queue.cc:1035] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb0c4d2e07d34e8fb08c10bed9ddcdbc" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:16.844113 22041 consensus_queue.cc:1035] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2fd1fbbad9b4634a318b2f5f3f1984c" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:16.846575 22050 mvcc.cc:205] Tried to move back new op lower bound from 7333619780989431808 to 7333619780925816832. Current Snapshot: MvccSnapshot[applied={T|T < 7333619780989431808}]
I20260926 14:55:16.846869 22049 mvcc.cc:205] Tried to move back new op lower bound from 7333619780989431808 to 7333619780925816832. Current Snapshot: MvccSnapshot[applied={T|T < 7333619780989431808}]
I20260926 14:55:16.848245 21562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:16.848328 21562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:16.849051 21562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8481001a22664d008f54138c1d2eb651: Sending DeleteTablet for 3 replicas of tablet 5f8a3c7fc51541139bdbbaab958c767e
I20260926 14:55:16.849340 21955 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8a3c7fc51541139bdbbaab958c767e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:16 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:49176
I20260926 14:55:16.849368 21821 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8a3c7fc51541139bdbbaab958c767e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:16 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:35804
I20260926 14:55:16.849472 21687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8a3c7fc51541139bdbbaab958c767e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:16 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:41908
I20260926 14:55:16.849608 22058 tablet_replica.cc:330] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: stopping tablet replica
I20260926 14:55:16.849696 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21624
I20260926 14:55:16.849716 22058 raft_consensus.cc:2229] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:55:16.849831 22058 raft_consensus.cc:2258] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:16.852381 22058 ts_tablet_manager.cc:1905] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:16.854494 22058 ts_tablet_manager.cc:1918] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:16.854566 22058 log.cc:1196] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/5f8a3c7fc51541139bdbbaab958c767e
I20260926 14:55:16.854856 22058 ts_tablet_manager.cc:1939] T 5f8a3c7fc51541139bdbbaab958c767e P 95e2ba1f02394f1cb24f903cfce819f2: Deleting consensus metadata
I20260926 14:55:16.854852 22059 tablet_replica.cc:330] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc: stopping tablet replica
I20260926 14:55:16.855017 22059 raft_consensus.cc:2229] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:16.855170 22059 raft_consensus.cc:2258] T 5f8a3c7fc51541139bdbbaab958c767e P bb0c4d2e07d34e8fb08c10bed9ddcdbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:16.855249 21546 catalog_manager.cc:4841] TS 95e2ba1f02394f1cb24f903cfce819f2 (127.5.15.131:35289): tablet 5f8a3c7fc51541139bdbbaab958c767e (table test-table [id=c901734fed24492e8f9613112038af65]) successfully deleted
W20260926 14:55:16.855456 21547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:36747 (error 108)
I20260926 14:55:16.855674 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21758
I20260926 14:55:16.856249 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.855786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36747 (local address 127.0.0.1:41916)
0926 14:55:16.856177 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.856212 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:36747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
W20260926 14:55:16.856364 21547 catalog_manager.cc:4584] TS c2fd1fbbad9b4634a318b2f5f3f1984c (127.5.15.129:36747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f8a3c7fc51541139bdbbaab958c767e: Network error: Client connection negotiation failed: client connection to 127.5.15.129:36747: connect: Connection refused (error 111)
I20260926 14:55:16.860491 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21892
I20260926 14:55:16.860634 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:16.860511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:40733 (local address 127.0.0.1:35812)
0926 14:55:16.860573 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:16.860601 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.130:40733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:55:16.865201 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 21530
2026-09-26T14:55:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1884 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T14:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:16Z Disabled control of system clock
I20260926 14:55:16.891422 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:33581
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:33581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:16.969321 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:16.969513 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:16.969534 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:16.970592 22065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:16.970633 22065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:16.970645 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:16.970657 22065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:16.970669 22065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:16.971992 22065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:33581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:16.972167 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:16.972371 22065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:16.976929 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.977031 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:16.977144 22070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:16.977205 22065 server_base.cc:1034] running on GCE node
I20260926 14:55:16.977444 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:16.977726 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:16.978889 22065 hybrid_clock.cc:648] HybridClock initialized: now 1790434516978887 us; error 40 us; skew 500 ppm
I20260926 14:55:16.979995 22065 webserver.cc:460] Webserver started at http://127.5.15.190:38775/ using document root <none> and password file <none>
I20260926 14:55:16.980240 22065 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:16.980305 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:16.980434 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:16.981413 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "9b5a016ffcaa4d46a6cd64e378882589"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "b58c37989594ed9d6ef8d538445f9e82"
server_key_iv: "fb9fcea79c34719554a60310fd0a9d92"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.981770 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "9b5a016ffcaa4d46a6cd64e378882589"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "b58c37989594ed9d6ef8d538445f9e82"
server_key_iv: "fb9fcea79c34719554a60310fd0a9d92"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.983008 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:16.983675 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:16.983852 22065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:16.983948 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "9b5a016ffcaa4d46a6cd64e378882589"
format_stamp: "Formatted at 2026-09-26 14:55:16 on dist-test-slave-s4p1"
server_key: "b58c37989594ed9d6ef8d538445f9e82"
server_key_iv: "fb9fcea79c34719554a60310fd0a9d92"
server_key_version: "encryptionkey@0"
I20260926 14:55:16.984027 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:17.006886 22065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:17.007179 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:17.007323 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:17.011150 22065 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:33581
I20260926 14:55:17.011189 22131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:33581 every 8 connection(s)
I20260926 14:55:17.011511 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:17.012382 22132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:17.014518 22132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: Bootstrap starting.
I20260926 14:55:17.015122 22132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:17.015416 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22065
I20260926 14:55:17.015466 22132 log.cc:824] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:17.015511 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:17.015734 5182 external_mini_cluster.cc:1428] Setting key 9fa61db2bfbec7b744d2ff126e75b4a8
I20260926 14:55:17.016256 22132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: No bootstrap required, opened a new log
I20260926 14:55:17.019459 22132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5a016ffcaa4d46a6cd64e378882589" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33581 } }
I20260926 14:55:17.019682 22132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:17.019762 22132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5a016ffcaa4d46a6cd64e378882589, State: Initialized, Role: FOLLOWER
I20260926 14:55:17.019948 22132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5a016ffcaa4d46a6cd64e378882589" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33581 } }
I20260926 14:55:17.020066 22132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:17.020123 22132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:17.020177 22132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:17.020835 22132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5a016ffcaa4d46a6cd64e378882589" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33581 } }
I20260926 14:55:17.020988 22132 leader_election.cc:302] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b5a016ffcaa4d46a6cd64e378882589; no voters:
I20260926 14:55:17.021160 22132 leader_election.cc:288] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:17.021211 22137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:17.021456 22137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [term 1 LEADER]: Becoming Leader. State: Replica: 9b5a016ffcaa4d46a6cd64e378882589, State: Running, Role: LEADER
I20260926 14:55:17.021476 22132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:17.021648 22137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5a016ffcaa4d46a6cd64e378882589" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33581 } }
I20260926 14:55:17.022174 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b5a016ffcaa4d46a6cd64e378882589. Latest consensus state: current_term: 1 leader_uuid: "9b5a016ffcaa4d46a6cd64e378882589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5a016ffcaa4d46a6cd64e378882589" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33581 } } }
I20260926 14:55:17.022302 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:17.023389 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b5a016ffcaa4d46a6cd64e378882589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5a016ffcaa4d46a6cd64e378882589" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33581 } } }
I20260926 14:55:17.023522 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:17.024199 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.016222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.5.15.190:33581)
0926 14:55:17.016487 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.016497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.016636 (+ 139us) server_negotiation.cc:407] Connection header received
0926 14:55:17.017935 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.017944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.018019 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.018191 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:17.018430 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.019850 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.021455 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.022100 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.022208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.023401 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:17.023404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:17.023405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:17.023603 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:17.023648 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:17.023662 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:17.023707 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.023852 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.023938 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
W20260926 14:55:17.024026 22152 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:17.024324 22152 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:17.024549 22146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:17.024698 22146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:17.026212 22146 catalog_manager.cc:1347] Generated new cluster ID: 5d6cad78bce34d8495edbd1eec22efe9
I20260926 14:55:17.026268 22146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:17.045030 22146 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:17.045734 22146 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:17.055972 22146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: Generated new TSK 0
I20260926 14:55:17.056143 22146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:17.066035 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:33581
--builtin_ntp_servers=127.5.15.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:17.143787 22156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:17.144012 22156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:17.144047 22156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:17.145190 22156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:17.145249 22156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:17.145289 22156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:55:17.146602 22156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:17.146898 22156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:17.147174 22156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:17.151985 22164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:17.151986 22161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:17.152024 22162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:17.152385 22156 server_base.cc:1034] running on GCE node
I20260926 14:55:17.152566 22156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:17.152745 22156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:17.153879 22156 hybrid_clock.cc:648] HybridClock initialized: now 1790434517153862 us; error 28 us; skew 500 ppm
I20260926 14:55:17.155045 22156 webserver.cc:460] Webserver started at http://127.5.15.129:43765/ using document root <none> and password file <none>
I20260926 14:55:17.155247 22156 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:17.155287 22156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:17.155369 22156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:17.156297 22156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "4b225d3b93714b8fb09c582597126090"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "ec074af4369c4ae0f013cdd214961912"
server_key_iv: "3ca134cab2df86f28fcba33c1ae25f59"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.156574 22156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "4b225d3b93714b8fb09c582597126090"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "ec074af4369c4ae0f013cdd214961912"
server_key_iv: "3ca134cab2df86f28fcba33c1ae25f59"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.157711 22156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:55:17.158397 22170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.158555 22156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:17.158622 22156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "4b225d3b93714b8fb09c582597126090"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "ec074af4369c4ae0f013cdd214961912"
server_key_iv: "3ca134cab2df86f28fcba33c1ae25f59"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.158668 22156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:17.171233 22156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:17.171444 22156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:17.171538 22156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:17.171703 22156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:17.171998 22156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:17.172029 22156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.172078 22156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:17.172102 22156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.177729 22156 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:42011
I20260926 14:55:17.177802 22283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:42011 every 8 connection(s)
I20260926 14:55:17.178099 22156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:55:17.180517 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22156
I20260926 14:55:17.180579 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:55:17.180763 5182 external_mini_cluster.cc:1428] Setting key c62d60de1cb660cada39e7f83ebc3338
I20260926 14:55:17.181581 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:33581
--builtin_ntp_servers=127.5.15.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:55:17.183404 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.178904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:44397 (local address 127.5.15.190:33581)
0926 14:55:17.178961 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.178963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.179205 (+ 242us) server_negotiation.cc:407] Connection header received
0926 14:55:17.179611 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.179611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.179624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.179677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:17.180325 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.180948 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.182026 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.182237 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.182320 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.183193 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:17.183194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:17.183195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:17.183198 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:17.183212 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:17.183216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:17.183259 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.183332 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.183350 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:55:17.183673 22286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.178913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:33581 (local address 127.5.15.129:44397)
0926 14:55:17.179075 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.179109 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.179534 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.179723 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.179726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.179734 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:17.180208 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.180212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.181039 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.181040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.181940 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.181943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.182333 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.182334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.182464 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.183020 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:17.183020 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:17.183057 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:17.183083 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:17.183261 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:17.183264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:17.183267 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.183336 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.183440 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260926 14:55:17.184080 22284 heartbeater.cc:347] Connected to a master server at 127.5.15.190:33581
I20260926 14:55:17.184170 22284 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:17.184342 22284 heartbeater.cc:511] Master 127.5.15.190:33581 requested a full tablet report, sending...
I20260926 14:55:17.184831 22096 ts_manager.cc:194] Registered new tserver with Master: 4b225d3b93714b8fb09c582597126090 (127.5.15.129:42011)
I20260926 14:55:17.185412 22096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.129:44397
W20260926 14:55:17.260293 22287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:17.260496 22287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:17.260532 22287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:17.261709 22287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:17.261771 22287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:17.261811 22287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:55:17.263243 22287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:17.263513 22287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:17.263783 22287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:17.268608 22293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:17.268612 22295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:17.268627 22292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:17.268823 22287 server_base.cc:1034] running on GCE node
I20260926 14:55:17.269009 22287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:17.269236 22287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:17.270426 22287 hybrid_clock.cc:648] HybridClock initialized: now 1790434517270415 us; error 37 us; skew 500 ppm
I20260926 14:55:17.271595 22287 webserver.cc:460] Webserver started at http://127.5.15.130:43677/ using document root <none> and password file <none>
I20260926 14:55:17.271832 22287 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:17.271899 22287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:17.272043 22287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:17.273020 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "1f68ce1ee4734c4da02024d4856e81c6"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "b15cdd56c97868171fc30290003a2c6a"
server_key_iv: "b07e1fffac61c208cc69447b4020beba"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.273355 22287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "1f68ce1ee4734c4da02024d4856e81c6"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "b15cdd56c97868171fc30290003a2c6a"
server_key_iv: "b07e1fffac61c208cc69447b4020beba"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.274559 22287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:17.275332 22301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.275489 22287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.275578 22287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "1f68ce1ee4734c4da02024d4856e81c6"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "b15cdd56c97868171fc30290003a2c6a"
server_key_iv: "b07e1fffac61c208cc69447b4020beba"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.275658 22287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:17.299682 22287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:17.299962 22287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:17.300103 22287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:17.300309 22287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:17.300604 22287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:17.300657 22287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.300699 22287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:17.300727 22287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.306160 22287 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:43009
I20260926 14:55:17.306563 22287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:55:17.306618 22414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:43009 every 8 connection(s)
I20260926 14:55:17.311244 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.307270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:48251 (local address 127.5.15.190:33581)
0926 14:55:17.307327 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.307329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.307597 (+ 268us) server_negotiation.cc:407] Connection header received
0926 14:55:17.308034 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.308035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.308044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.308096 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:17.308698 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.309173 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.310049 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.310222 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.310270 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.311021 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:17.311023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:17.311023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:17.311027 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:17.311042 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:17.311045 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:17.311088 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.311168 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.311187 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:55:17.311455 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.307265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:33581 (local address 127.5.15.130:48251)
0926 14:55:17.307441 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.307495 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.307954 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.308162 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.308165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.308172 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:17.308635 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.308638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.309248 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.309249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.309982 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.309983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.310289 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.310290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.310390 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.310902 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:17.310902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:17.310927 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:17.310947 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:17.311099 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:17.311103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:17.311105 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.311153 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.311233 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260926 14:55:17.311906 22415 heartbeater.cc:347] Connected to a master server at 127.5.15.190:33581
I20260926 14:55:17.312011 22415 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:17.312184 22415 heartbeater.cc:511] Master 127.5.15.190:33581 requested a full tablet report, sending...
I20260926 14:55:17.312542 22096 ts_manager.cc:194] Registered new tserver with Master: 1f68ce1ee4734c4da02024d4856e81c6 (127.5.15.130:43009)
I20260926 14:55:17.312862 22096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.130:48251
I20260926 14:55:17.316614 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22287
I20260926 14:55:17.316690 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:55:17.316828 5182 external_mini_cluster.cc:1428] Setting key 9b76f77ce352423d35e928ba2a100640
I20260926 14:55:17.317641 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:33581
--builtin_ntp_servers=127.5.15.148:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:17.394339 22418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:17.394567 22418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:17.394601 22418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:17.395769 22418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:17.395834 22418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:17.395874 22418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:55:17.397154 22418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:17.397405 22418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:17.397679 22418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:17.402380 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:17.402426 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:17.402366 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:17.402462 22418 server_base.cc:1034] running on GCE node
I20260926 14:55:17.402828 22418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:17.403066 22418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:17.404204 22418 hybrid_clock.cc:648] HybridClock initialized: now 1790434517404194 us; error 28 us; skew 500 ppm
I20260926 14:55:17.405364 22418 webserver.cc:460] Webserver started at http://127.5.15.131:36993/ using document root <none> and password file <none>
I20260926 14:55:17.405587 22418 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:17.405629 22418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:17.405761 22418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:17.406795 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "c99049a7905b691f8eb47ac5bb5da27e"
server_key_iv: "d9d38d6104407a118012617137dba98f"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.407133 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "c99049a7905b691f8eb47ac5bb5da27e"
server_key_iv: "d9d38d6104407a118012617137dba98f"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.408373 22418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:17.409055 22432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.409212 22418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:17.409307 22418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c"
format_stamp: "Formatted at 2026-09-26 14:55:17 on dist-test-slave-s4p1"
server_key: "c99049a7905b691f8eb47ac5bb5da27e"
server_key_iv: "d9d38d6104407a118012617137dba98f"
server_key_version: "encryptionkey@0"
I20260926 14:55:17.409384 22418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:17.430931 22418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:17.431202 22418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:17.431362 22418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:17.431576 22418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:17.431888 22418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:17.431943 22418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.431984 22418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:17.432010 22418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:17.437497 22418 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:41079
I20260926 14:55:17.437536 22545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:41079 every 8 connection(s)
I20260926 14:55:17.437873 22418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:55:17.442507 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22418
I20260926 14:55:17.442606 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:55:17.442844 5182 external_mini_cluster.cc:1428] Setting key e3ba638dba714335a49e50ef91778854
I20260926 14:55:17.442904 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.438633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:38395 (local address 127.5.15.190:33581)
0926 14:55:17.438676 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.438677 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.438939 (+ 262us) server_negotiation.cc:407] Connection header received
0926 14:55:17.439393 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.439394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.439404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.439454 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:17.440073 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.440540 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.441506 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.441680 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.441733 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.442562 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:17.442564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:17.442564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:17.442569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:17.442583 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:17.442588 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:17.442640 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.442790 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.442814 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:55:17.443136 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.438618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:33581 (local address 127.5.15.131:38395)
0926 14:55:17.438822 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.438860 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.439312 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.439511 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.439514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.439522 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:17.440007 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.440011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.440618 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.440619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.441420 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.441422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.441753 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.441754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.441862 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.442418 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:17.442418 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:17.442444 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:17.442464 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:17.442675 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:17.442680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:17.442684 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.442794 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.442898 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260926 14:55:17.443591 22546 heartbeater.cc:347] Connected to a master server at 127.5.15.190:33581
I20260926 14:55:17.443703 22546 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:17.443882 22546 heartbeater.cc:511] Master 127.5.15.190:33581 requested a full tablet report, sending...
I20260926 14:55:17.444217 22096 ts_manager.cc:194] Registered new tserver with Master: 585d7cb5dc9f4b3d914ecae2b867fc7c (127.5.15.131:41079)
I20260926 14:55:17.444578 22096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.131:38395
I20260926 14:55:17.454777 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:55:17.459020 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.456080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.5.15.190:33581)
0926 14:55:17.456152 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.456154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.456244 (+ 90us) server_negotiation.cc:407] Connection header received
0926 14:55:17.456308 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.456309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.456319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.456381 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:17.456586 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.457053 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.457809 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.458004 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.458079 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.458794 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:17.458796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:17.458796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:17.458800 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:17.458816 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:17.458819 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:17.458866 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.458916 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.458938 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:55:17.460464 22096 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:55:17.462080 22096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:55:17.468231 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.464119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42011 (local address 127.0.0.1:41582)
0926 14:55:17.464294 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.464332 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.464401 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.464993 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.464994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.464997 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:17.465046 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.465048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.465948 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.465950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.467204 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.467207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.467950 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.467951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.468026 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.468039 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.468110 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.468161 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260926 14:55:17.468243 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.464206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.5.15.129:42011)
0926 14:55:17.464807 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.464812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.464822 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:17.464875 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.464877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.464893 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.465138 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:17.465200 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.465863 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.467301 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.467846 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.467946 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.468046 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.468121 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.468157 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":84,"threads_started":1}
I20260926 14:55:17.468456 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.464122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:41079 (local address 127.0.0.1:44230)
0926 14:55:17.464373 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.464383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.464425 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.464716 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.464719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.464721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:17.464790 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.464792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.466298 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.466299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.467206 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.467208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.468209 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.468210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.468277 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.468285 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.468336 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.468386 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":10,"thread_start_us":99,"threads_started":1}
I20260926 14:55:17.468703 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.464357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.5.15.131:41079)
0926 14:55:17.464522 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.464528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.464540 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:55:17.464580 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.464581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.464598 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.464690 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:17.464884 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.466214 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.467292 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.468014 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.468084 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.468175 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.468558 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.468601 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":109,"threads_started":1}
I20260926 14:55:17.468775 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.464127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:43009 (local address 127.0.0.1:36016)
0926 14:55:17.464504 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.464512 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.464553 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.464947 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.464948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.464950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:17.464982 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.464983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.466281 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.466282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.467361 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.467362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.468214 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.468214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.468277 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.468630 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.468665 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.468705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":29,"thread_start_us":99,"threads_started":1}
I20260926 14:55:17.469139 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.464570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.5.15.130:43009)
0926 14:55:17.464773 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.464778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.464788 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:17.464827 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.464829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.464847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.465049 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:17.465104 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.466190 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.467439 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.468113 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.468872 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.468990 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.469017 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.469059 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260926 14:55:17.469079 22218 tablet_service.cc:1467] Processing CreateTablet for tablet 86df3bfbc02a4e3c945b8e10308d14ca (DEFAULT_TABLE table=test-table [id=d160a85da26b4873b08e284080dd1109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:17.469466 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86df3bfbc02a4e3c945b8e10308d14ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:17.469390 22480 tablet_service.cc:1467] Processing CreateTablet for tablet 86df3bfbc02a4e3c945b8e10308d14ca (DEFAULT_TABLE table=test-table [id=d160a85da26b4873b08e284080dd1109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:17.469662 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86df3bfbc02a4e3c945b8e10308d14ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:17.469758 22349 tablet_service.cc:1467] Processing CreateTablet for tablet 86df3bfbc02a4e3c945b8e10308d14ca (DEFAULT_TABLE table=test-table [id=d160a85da26b4873b08e284080dd1109]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:17.470067 22349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86df3bfbc02a4e3c945b8e10308d14ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:17.472024 22560 tablet_bootstrap.cc:492] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Bootstrap starting.
I20260926 14:55:17.472671 22560 tablet_bootstrap.cc:654] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:17.472853 22561 tablet_bootstrap.cc:492] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090: Bootstrap starting.
I20260926 14:55:17.472997 22560 log.cc:824] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:17.473284 22562 tablet_bootstrap.cc:492] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Bootstrap starting.
I20260926 14:55:17.473471 22561 tablet_bootstrap.cc:654] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:17.473770 22561 log.cc:824] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:17.473932 22562 tablet_bootstrap.cc:654] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:17.473968 22560 tablet_bootstrap.cc:492] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: No bootstrap required, opened a new log
I20260926 14:55:17.474020 22560 ts_tablet_manager.cc:1397] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:17.474275 22562 log.cc:824] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:17.474606 22561 tablet_bootstrap.cc:492] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090: No bootstrap required, opened a new log
I20260926 14:55:17.474682 22561 ts_tablet_manager.cc:1397] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:17.475008 22562 tablet_bootstrap.cc:492] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: No bootstrap required, opened a new log
I20260926 14:55:17.475070 22562 ts_tablet_manager.cc:1397] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:17.476159 22560 raft_consensus.cc:348] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.476325 22560 raft_consensus.cc:374] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:17.476373 22560 raft_consensus.cc:729] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585d7cb5dc9f4b3d914ecae2b867fc7c, State: Initialized, Role: FOLLOWER
I20260926 14:55:17.476478 22560 consensus_queue.cc:260] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.476738 22560 ts_tablet_manager.cc:1428] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:17.476917 22546 heartbeater.cc:503] Master 127.5.15.190:33581 was elected leader, sending a full tablet report...
I20260926 14:55:17.477317 22561 raft_consensus.cc:348] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.477465 22561 raft_consensus.cc:374] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:17.477501 22561 raft_consensus.cc:729] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b225d3b93714b8fb09c582597126090, State: Initialized, Role: FOLLOWER
I20260926 14:55:17.477530 22562 raft_consensus.cc:348] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.477619 22561 consensus_queue.cc:260] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.477667 22562 raft_consensus.cc:374] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:17.477711 22562 raft_consensus.cc:729] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f68ce1ee4734c4da02024d4856e81c6, State: Initialized, Role: FOLLOWER
I20260926 14:55:17.477880 22561 ts_tablet_manager.cc:1428] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 14:55:17.477854 22562 consensus_queue.cc:260] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.477999 22284 heartbeater.cc:503] Master 127.5.15.190:33581 was elected leader, sending a full tablet report...
I20260926 14:55:17.478096 22562 ts_tablet_manager.cc:1428] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 14:55:17.478143 22415 heartbeater.cc:503] Master 127.5.15.190:33581 was elected leader, sending a full tablet report...
I20260926 14:55:17.482142 22568 raft_consensus.cc:482] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:17.482219 22568 raft_consensus.cc:504] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.482553 22568 leader_election.cc:288] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585d7cb5dc9f4b3d914ecae2b867fc7c (127.5.15.131:41079), 4b225d3b93714b8fb09c582597126090 (127.5.15.129:42011)
I20260926 14:55:17.485307 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.482843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:57211 (local address 127.5.15.131:41079)
0926 14:55:17.482885 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.482887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.482934 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:55:17.482997 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.482998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.483024 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.483076 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:17.483271 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.483742 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.484638 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.484984 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.485061 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.485069 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.485216 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.485247 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:55:17.485321 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.482698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:41079 (local address 127.5.15.130:57211)
0926 14:55:17.482874 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.482885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.482945 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.483077 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.483078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.483079 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:17.483193 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.483195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.483829 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.483830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.484545 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.484547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.485060 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.485061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.485130 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.485134 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.485209 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.485245 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260926 14:55:17.485370 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.482893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:41213 (local address 127.5.15.129:42011)
0926 14:55:17.482931 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.482933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.482940 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:55:17.482961 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.482962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.482973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.483027 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:17.483189 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.483676 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.484595 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.485026 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.485104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.485111 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.485265 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.485296 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:55:17.485622 22500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86df3bfbc02a4e3c945b8e10308d14ca" candidate_uuid: "1f68ce1ee4734c4da02024d4856e81c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" is_pre_election: true
I20260926 14:55:17.485721 22500 raft_consensus.cc:2454] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f68ce1ee4734c4da02024d4856e81c6 in term 0.
I20260926 14:55:17.485915 22305 leader_election.cc:302] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f68ce1ee4734c4da02024d4856e81c6, 585d7cb5dc9f4b3d914ecae2b867fc7c; no voters:
I20260926 14:55:17.486006 22417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.482688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42011 (local address 127.5.15.130:41213)
0926 14:55:17.482737 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.482768 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:17.482809 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:17.483045 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:17.483046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:17.483048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:17.483115 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.483116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.483761 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.483762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.484503 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:17.484505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.485103 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.485104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:17.485173 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.485176 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:17.485257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:55:17.485297 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:55:17.486023 22568 raft_consensus.cc:2781] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:55:17.486248 22568 raft_consensus.cc:482] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:55:17.486291 22568 raft_consensus.cc:3037] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:17.486311 22238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86df3bfbc02a4e3c945b8e10308d14ca" candidate_uuid: "1f68ce1ee4734c4da02024d4856e81c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b225d3b93714b8fb09c582597126090" is_pre_election: true
I20260926 14:55:17.486399 22238 raft_consensus.cc:2454] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f68ce1ee4734c4da02024d4856e81c6 in term 0.
I20260926 14:55:17.487028 22568 raft_consensus.cc:504] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.487167 22568 leader_election.cc:288] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [CANDIDATE]: Term 1 election: Requested vote from peers 585d7cb5dc9f4b3d914ecae2b867fc7c (127.5.15.131:41079), 4b225d3b93714b8fb09c582597126090 (127.5.15.129:42011)
I20260926 14:55:17.487326 22500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86df3bfbc02a4e3c945b8e10308d14ca" candidate_uuid: "1f68ce1ee4734c4da02024d4856e81c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c"
I20260926 14:55:17.487407 22500 raft_consensus.cc:3037] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:17.488242 22500 raft_consensus.cc:2454] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f68ce1ee4734c4da02024d4856e81c6 in term 1.
I20260926 14:55:17.488365 22238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86df3bfbc02a4e3c945b8e10308d14ca" candidate_uuid: "1f68ce1ee4734c4da02024d4856e81c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b225d3b93714b8fb09c582597126090"
I20260926 14:55:17.488411 22305 leader_election.cc:302] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f68ce1ee4734c4da02024d4856e81c6, 585d7cb5dc9f4b3d914ecae2b867fc7c; no voters:
I20260926 14:55:17.488438 22238 raft_consensus.cc:3037] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:17.489046 22238 raft_consensus.cc:2454] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f68ce1ee4734c4da02024d4856e81c6 in term 1.
I20260926 14:55:17.489126 22568 raft_consensus.cc:2781] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:17.489276 22568 raft_consensus.cc:686] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 1 LEADER]: Becoming Leader. State: Replica: 1f68ce1ee4734c4da02024d4856e81c6, State: Running, Role: LEADER
I20260926 14:55:17.489384 22568 consensus_queue.cc:237] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } }
I20260926 14:55:17.490091 22096 catalog_manager.cc:5495] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f68ce1ee4734c4da02024d4856e81c6 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "1f68ce1ee4734c4da02024d4856e81c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f68ce1ee4734c4da02024d4856e81c6" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43009 } health_report { overall_health: HEALTHY } } }
I20260926 14:55:17.498075 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.495868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.5.15.130:43009)
0926 14:55:17.495910 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:17.495912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:17.495921 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:17.495978 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:17.495979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:17.495995 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:17.496054 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:17.496177 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.496602 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.497307 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:17.497492 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:17.497578 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:17.497739 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:17.497832 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:17.497931 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:17.497972 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:55:17.497990 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:55:17.499679 22238 raft_consensus.cc:1261] T 86df3bfbc02a4e3c945b8e10308d14ca P 4b225d3b93714b8fb09c582597126090 [term 1 FOLLOWER]: Refusing update from remote peer 1f68ce1ee4734c4da02024d4856e81c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:17.499679 22500 raft_consensus.cc:1261] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 1 FOLLOWER]: Refusing update from remote peer 1f68ce1ee4734c4da02024d4856e81c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:17.500041 22570 consensus_queue.cc:1035] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585d7cb5dc9f4b3d914ecae2b867fc7c" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:17.500156 22568 consensus_queue.cc:1035] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b225d3b93714b8fb09c582597126090" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:17.503037 22574 mvcc.cc:205] Tried to move back new op lower bound from 7333619783676444672 to 7333619783637135360. Current Snapshot: MvccSnapshot[applied={T|T < 7333619783676444672}]
I20260926 14:55:17.503319 22575 mvcc.cc:205] Tried to move back new op lower bound from 7333619783676444672 to 7333619783637135360. Current Snapshot: MvccSnapshot[applied={T|T < 7333619783676444672 or (T in {7333619783676444672})}]
I20260926 14:55:17.504431 22096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:17.504516 22096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:17.505230 22096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b5a016ffcaa4d46a6cd64e378882589: Sending DeleteTablet for 3 replicas of tablet 86df3bfbc02a4e3c945b8e10308d14ca
I20260926 14:55:17.505534 22480 tablet_service.cc:1514] Processing DeleteTablet for tablet 86df3bfbc02a4e3c945b8e10308d14ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:17 UTC) from {username='slave'} at 127.0.0.1:44230
I20260926 14:55:17.505558 22218 tablet_service.cc:1514] Processing DeleteTablet for tablet 86df3bfbc02a4e3c945b8e10308d14ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:17 UTC) from {username='slave'} at 127.0.0.1:41582
I20260926 14:55:17.505745 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22156
I20260926 14:55:17.506217 22349 tablet_service.cc:1514] Processing DeleteTablet for tablet 86df3bfbc02a4e3c945b8e10308d14ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:17 UTC) from {username='slave'} at 127.0.0.1:36016
I20260926 14:55:17.506453 22583 tablet_replica.cc:330] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: stopping tablet replica
I20260926 14:55:17.506534 22583 raft_consensus.cc:2229] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:17.506620 22583 raft_consensus.cc:2258] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:17.507341 22583 ts_tablet_manager.cc:1905] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:17.508718 22583 ts_tablet_manager.cc:1918] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:17.508797 22583 log.cc:1196] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/86df3bfbc02a4e3c945b8e10308d14ca
I20260926 14:55:17.509027 22583 ts_tablet_manager.cc:1939] T 86df3bfbc02a4e3c945b8e10308d14ca P 585d7cb5dc9f4b3d914ecae2b867fc7c: Deleting consensus metadata
I20260926 14:55:17.509388 22083 catalog_manager.cc:4841] TS 585d7cb5dc9f4b3d914ecae2b867fc7c (127.5.15.131:41079): tablet 86df3bfbc02a4e3c945b8e10308d14ca (table test-table [id=d160a85da26b4873b08e284080dd1109]) successfully deleted
I20260926 14:55:17.510855 22585 tablet_replica.cc:330] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: stopping tablet replica
I20260926 14:55:17.510957 22585 raft_consensus.cc:2229] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:55:17.511063 22585 raft_consensus.cc:2258] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:17.511461 22585 ts_tablet_manager.cc:1905] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:55:17.512562 22082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:42011 (error 108)
I20260926 14:55:17.512989 22585 ts_tablet_manager.cc:1918] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:17.513052 22585 log.cc:1196] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/86df3bfbc02a4e3c945b8e10308d14ca
I20260926 14:55:17.513083 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.512952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42011 (local address 127.0.0.1:41598)
0926 14:55:17.512996 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.513052 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:42011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 14:55:17.513099 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22287
I20260926 14:55:17.513284 22585 ts_tablet_manager.cc:1939] T 86df3bfbc02a4e3c945b8e10308d14ca P 1f68ce1ee4734c4da02024d4856e81c6: Deleting consensus metadata
W20260926 14:55:17.513823 22082 catalog_manager.cc:4584] TS 4b225d3b93714b8fb09c582597126090 (127.5.15.129:42011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86df3bfbc02a4e3c945b8e10308d14ca: Network error: Client connection negotiation failed: client connection to 127.5.15.129:42011: connect: Connection refused (error 111)
I20260926 14:55:17.518532 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22418
I20260926 14:55:17.518594 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:17.518478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:43009 (local address 127.0.0.1:36040)
0926 14:55:17.518535 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:17.518564 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.130:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:55:17.523000 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22065
2026-09-26T14:55:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (657 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:17 dist-test-slave-s4p1 krb5kdc[22589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:17 dist-test-slave-s4p1 krb5kdc[22589](info): set up 2 sockets
Sep 26 14:55:17 dist-test-slave-s4p1 krb5kdc[22589](info): commencing operation
krb5kdc: starting...
W20260926 14:55:18.546397 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:18 dist-test-slave-s4p1 krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:18.655761 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44343
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:44343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:18.734300 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:18.734539 22602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:18.734572 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:18.735682 22602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:18.735746 22602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:18.735769 22602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:18.735788 22602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:18.735817 22602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:18.737251 22602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:44343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:44343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:18.737521 22602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:18.737731 22602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:18.742251 22607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:18.742272 22610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:18.742278 22608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:18.742383 22602 server_base.cc:1034] running on GCE node
I20260926 14:55:18.742702 22602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:18.742991 22602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:18.744150 22602 hybrid_clock.cc:648] HybridClock initialized: now 1790434518744132 us; error 29 us; skew 500 ppm
Sep 26 14:55:18 dist-test-slave-s4p1 krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:18.746471 22602 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:18.746971 22602 webserver.cc:460] Webserver started at http://127.5.15.190:39641/ using document root <none> and password file <none>
I20260926 14:55:18.747231 22602 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:18.747298 22602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:18.747454 22602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:18.748435 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "e0758f649d164f7687bebd65546715a1"
format_stamp: "Formatted at 2026-09-26 14:55:18 on dist-test-slave-s4p1"
server_key: "75ee820d5168c25d959056d0f1adac6a"
server_key_iv: "465dc8053fa0e55fff9c51ccb70f7645"
server_key_version: "encryptionkey@0"
I20260926 14:55:18.748821 22602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "e0758f649d164f7687bebd65546715a1"
format_stamp: "Formatted at 2026-09-26 14:55:18 on dist-test-slave-s4p1"
server_key: "75ee820d5168c25d959056d0f1adac6a"
server_key_iv: "465dc8053fa0e55fff9c51ccb70f7645"
server_key_version: "encryptionkey@0"
I20260926 14:55:18.750097 22602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:18.750809 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:18.750990 22602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:18.751104 22602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "e0758f649d164f7687bebd65546715a1"
format_stamp: "Formatted at 2026-09-26 14:55:18 on dist-test-slave-s4p1"
server_key: "75ee820d5168c25d959056d0f1adac6a"
server_key_iv: "465dc8053fa0e55fff9c51ccb70f7645"
server_key_version: "encryptionkey@0"
I20260926 14:55:18.751189 22602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:18.769312 22602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:18.771311 22602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:18.771476 22602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:18.775194 22602 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:44343
I20260926 14:55:18.775228 22669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:44343 every 8 connection(s)
I20260926 14:55:18.775568 22602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:18.776373 22670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:18.778424 22670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1: Bootstrap starting.
I20260926 14:55:18.778947 22670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:18.779170 22670 log.cc:824] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:18.779783 22670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1: No bootstrap required, opened a new log
I20260926 14:55:18.780200 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22602
I20260926 14:55:18.780315 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:18.780509 5182 external_mini_cluster.cc:1428] Setting key 5fc4a8277b42e877bfba7cfadb878640
I20260926 14:55:18.781525 22670 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0758f649d164f7687bebd65546715a1" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44343 } }
I20260926 14:55:18.782004 22670 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:18.782070 22670 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0758f649d164f7687bebd65546715a1, State: Initialized, Role: FOLLOWER
I20260926 14:55:18.782213 22670 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0758f649d164f7687bebd65546715a1" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44343 } }
I20260926 14:55:18.782300 22670 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:18.782335 22670 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:18.782383 22670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:18.782944 22670 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0758f649d164f7687bebd65546715a1" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44343 } }
I20260926 14:55:18.783025 22670 leader_election.cc:302] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0758f649d164f7687bebd65546715a1; no voters:
I20260926 14:55:18.783246 22670 leader_election.cc:288] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:18.783265 22675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:18.783387 22675 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [term 1 LEADER]: Becoming Leader. State: Replica: e0758f649d164f7687bebd65546715a1, State: Running, Role: LEADER
I20260926 14:55:18.783510 22670 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:18.783519 22675 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0758f649d164f7687bebd65546715a1" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44343 } }
I20260926 14:55:18.783936 22676 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0758f649d164f7687bebd65546715a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0758f649d164f7687bebd65546715a1" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44343 } } }
I20260926 14:55:18.784034 22676 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:18.783948 22677 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0758f649d164f7687bebd65546715a1. Latest consensus state: current_term: 1 leader_uuid: "e0758f649d164f7687bebd65546715a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0758f649d164f7687bebd65546715a1" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 44343 } } }
I20260926 14:55:18.784312 22677 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:18.784500 22683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:18.785004 22683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:18.786676 22683 catalog_manager.cc:1347] Generated new cluster ID: e1441fbb64f84c65bff77736e97e1304
I20260926 14:55:18.786741 22683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:55:18 dist-test-slave-s4p1 krb5kdc[22589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:18.791464 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:18.781007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.5.15.190:44343)
0926 14:55:18.781167 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:18.781175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:18.781206 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:55:18.781509 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:18.781515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:18.781558 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:18.781667 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:18.782475 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:18.783899 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:18.784979 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:18.785598 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:18.785732 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:18.788212 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:18.788215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:18.788216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:18.788227 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:18.789722 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:18.790096 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:18.790099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:18.790099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:18.790204 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:18.790429 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:18.790429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:18.790430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:18.790650 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:18.790846 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:18.790983 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 14:55:18.791120 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260926 14:55:18.791672 22683 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:18.792307 22683 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:18.799455 22683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1: Generated new TSK 0
I20260926 14:55:18.799623 22683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:55:18.824836 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44343
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:18.904531 22696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:18.904727 22696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:18.904750 22696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:18.905928 22696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:18.905970 22696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:18.905997 22696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:55:18.907416 22696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:18.907648 22696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:18.907878 22696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:18.912703 22704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:18.912714 22701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:18.912731 22702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:18.912919 22696 server_base.cc:1034] running on GCE node
I20260926 14:55:18.913128 22696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:18.913404 22696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:18.914582 22696 hybrid_clock.cc:648] HybridClock initialized: now 1790434518914569 us; error 24 us; skew 500 ppm
Sep 26 14:55:18 dist-test-slave-s4p1 krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:18.917027 22696 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:55:18.917394 22696 webserver.cc:460] Webserver started at http://127.5.15.129:46215/ using document root <none> and password file <none>
I20260926 14:55:18.917567 22696 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:18.917605 22696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:18.917689 22696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:18.918681 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "07ba639f446a45258f6b8a30680ee738"
format_stamp: "Formatted at 2026-09-26 14:55:18 on dist-test-slave-s4p1"
server_key: "0c9bd3b1c51518287218d6045d1d5989"
server_key_iv: "810acbd6fa5eb0e6f06d3872b9e0006f"
server_key_version: "encryptionkey@0"
I20260926 14:55:18.919062 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "07ba639f446a45258f6b8a30680ee738"
format_stamp: "Formatted at 2026-09-26 14:55:18 on dist-test-slave-s4p1"
server_key: "0c9bd3b1c51518287218d6045d1d5989"
server_key_iv: "810acbd6fa5eb0e6f06d3872b9e0006f"
server_key_version: "encryptionkey@0"
I20260926 14:55:18.920257 22696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:18.920912 22711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:18.921064 22696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:18.921150 22696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "07ba639f446a45258f6b8a30680ee738"
format_stamp: "Formatted at 2026-09-26 14:55:18 on dist-test-slave-s4p1"
server_key: "0c9bd3b1c51518287218d6045d1d5989"
server_key_iv: "810acbd6fa5eb0e6f06d3872b9e0006f"
server_key_version: "encryptionkey@0"
I20260926 14:55:18.921206 22696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:18.931764 22696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:18.933919 22696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:18.934021 22696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:18.934175 22696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:18.934419 22696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:18.934453 22696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:18.934473 22696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:18.934485 22696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:18.940044 22696 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:36935
I20260926 14:55:18.940117 22824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:36935 every 8 connection(s)
I20260926 14:55:18.940413 22696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:55:18 dist-test-slave-s4p1 krb5kdc[22589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:18.949108 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22696
I20260926 14:55:18.949220 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:55:18.949429 5182 external_mini_cluster.cc:1428] Setting key 26b1f99bef3f32025832fc2e773773a3
I20260926 14:55:18.949590 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:18.941152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:42683 (local address 127.5.15.190:44343)
0926 14:55:18.941212 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:18.941214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:18.941422 (+ 208us) server_negotiation.cc:407] Connection header received
0926 14:55:18.941982 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:18.941983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:18.941998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:18.942061 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:18.943059 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:18.943570 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:18.944576 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:18.944814 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:18.944906 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:18.946884 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:18.946886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:18.946886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:18.946893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:18.948309 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:18.948643 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:18.948644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:18.948644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:18.948700 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:18.948910 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:18.948910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:18.948910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:18.949006 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:18.949054 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:18.949386 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 14:55:18.949453 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:55:18.949782 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:18.941131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44343 (local address 127.5.15.129:42683)
0926 14:55:18.941296 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:18.941329 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:18.941879 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:18.942128 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:18.942132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:18.942715 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:18.942961 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:18.942966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:18.943666 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:18.943667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:18.944465 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:18.944467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:18.944919 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:18.944920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:18.945046 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:18.945442 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:18.945449 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:18.946791 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:18.948376 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:18.948381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:18.948382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:18.948582 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:18.948760 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:18.948760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:18.948760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:18.948856 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:18.949071 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:18.949078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:18.949245 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 14:55:18.949369 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 14:55:18.949532 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260926 14:55:18.950270 22825 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44343
I20260926 14:55:18.950368 22825 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:18.950551 22825 heartbeater.cc:511] Master 127.5.15.190:44343 requested a full tablet report, sending...
I20260926 14:55:18.951086 22633 ts_manager.cc:194] Registered new tserver with Master: 07ba639f446a45258f6b8a30680ee738 (127.5.15.129:36935)
I20260926 14:55:18.951944 22633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:42683
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:55:18.973073 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44343
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:19.049858 22830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:19.050055 22830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:19.050076 22830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:19.051172 22830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:19.051204 22830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:19.051246 22830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:55:19.053215 22830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:19.053393 22830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:19.053629 22830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:19.058254 22835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.058398 22830 server_base.cc:1034] running on GCE node
W20260926 14:55:19.058264 22836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.058293 22838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.058666 22830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:19.058934 22830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:19.060070 22830 hybrid_clock.cc:648] HybridClock initialized: now 1790434519060059 us; error 28 us; skew 500 ppm
Sep 26 14:55:19 dist-test-slave-s4p1 krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:19.062330 22830 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:55:19.062822 22830 webserver.cc:460] Webserver started at http://127.5.15.130:46477/ using document root <none> and password file <none>
I20260926 14:55:19.063041 22830 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:19.063086 22830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:19.063223 22830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:19.064153 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "f688fc4162544eb19cbd9a23776ca85f"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "08e599ecf120a0471648557558b9096f"
server_key_iv: "8e88c04c6091497e17afafa3058e6650"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.064486 22830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "f688fc4162544eb19cbd9a23776ca85f"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "08e599ecf120a0471648557558b9096f"
server_key_iv: "8e88c04c6091497e17afafa3058e6650"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.065721 22830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:19.066481 22845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.066689 22830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:19.066803 22830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "f688fc4162544eb19cbd9a23776ca85f"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "08e599ecf120a0471648557558b9096f"
server_key_iv: "8e88c04c6091497e17afafa3058e6650"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.066890 22830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:19.086374 22830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:19.088183 22830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:19.088327 22830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:19.088567 22830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:19.088881 22830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:19.088912 22830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.088933 22830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:19.088977 22830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.094623 22830 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:43767
I20260926 14:55:19.094659 22958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:43767 every 8 connection(s)
I20260926 14:55:19.095017 22830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:55:19.097149 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22830
I20260926 14:55:19.097258 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:55:19.097443 5182 external_mini_cluster.cc:1428] Setting key 22cfb3c6db0a8a6d3c627f5f72932345
Sep 26 14:55:19 dist-test-slave-s4p1 krb5kdc[22589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:19.105639 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.095766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:58753 (local address 127.5.15.190:44343)
0926 14:55:19.095813 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.095815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.096014 (+ 199us) server_negotiation.cc:407] Connection header received
0926 14:55:19.096462 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.096463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.096472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.096527 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:19.097660 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.098439 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.100010 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.100329 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.100404 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.102846 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:19.102848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:19.102848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:19.102855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:19.104255 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:19.104729 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:19.104730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:19.104730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:19.104779 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:19.105052 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:19.105053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:19.105053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:19.105148 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:19.105197 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.105512 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.105576 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:55:19.105917 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.095730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44343 (local address 127.5.15.130:58753)
0926 14:55:19.095881 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.095921 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.096379 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.096577 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.096580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.097272 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:19.097556 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.097564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.098524 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.098525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.099911 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.099914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.100412 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.100412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.100508 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.101033 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:19.101041 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:19.102735 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:19.104339 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:19.104347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:19.104350 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:19.104650 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:19.104845 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:19.104846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:19.104846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:19.104984 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:19.105213 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:19.105219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:19.105373 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.105494 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.105660 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260926 14:55:19.106447 22959 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44343
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
I20260926 14:55:19.106528 22959 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:19.106700 22959 heartbeater.cc:511] Master 127.5.15.190:44343 requested a full tablet report, sending...
I20260926 14:55:19.107101 22633 ts_manager.cc:194] Registered new tserver with Master: f688fc4162544eb19cbd9a23776ca85f (127.5.15.130:43767)
I20260926 14:55:19.107417 22633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:58753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:55:19.121464 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44343
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:43101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:19.200516 22964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:19.200747 22964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:19.200783 22964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:19.201920 22964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:19.201987 22964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:19.202030 22964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:55:19.203426 22964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:43101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:44343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:19.203684 22964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:19.203910 22964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:19.208803 22972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.208825 22969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.208954 22970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.208834 22964 server_base.cc:1034] running on GCE node
I20260926 14:55:19.209252 22964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:19.209496 22964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:19.210668 22964 hybrid_clock.cc:648] HybridClock initialized: now 1790434519210647 us; error 32 us; skew 500 ppm
Sep 26 14:55:19 dist-test-slave-s4p1 krb5kdc[22589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:19.213124 22964 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:55:19.213541 22964 webserver.cc:460] Webserver started at http://127.5.15.131:35167/ using document root <none> and password file <none>
I20260926 14:55:19.213707 22964 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:19.213745 22964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:19.213828 22964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:19.214730 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "72f08b704f454cde971bb8437217609e"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "e3d22ba5bb6dd8ae82c3277c235bf1c4"
server_key_iv: "f9be388d3dcd12736037df8da62f93b1"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.215077 22964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "72f08b704f454cde971bb8437217609e"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "e3d22ba5bb6dd8ae82c3277c235bf1c4"
server_key_iv: "f9be388d3dcd12736037df8da62f93b1"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.216269 22964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:19.216877 22979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.217069 22964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:19.217134 22964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "72f08b704f454cde971bb8437217609e"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "e3d22ba5bb6dd8ae82c3277c235bf1c4"
server_key_iv: "f9be388d3dcd12736037df8da62f93b1"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.217226 22964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:19.225176 22964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:19.227737 22964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:19.227873 22964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:19.228085 22964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:19.228427 22964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:19.228477 22964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.228507 22964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:19.228538 22964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.233932 22964 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:46845
I20260926 14:55:19.233997 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:46845 every 8 connection(s)
I20260926 14:55:19.234373 22964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:55:19.235695 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 22964
I20260926 14:55:19.235817 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:55:19.236112 5182 external_mini_cluster.cc:1428] Setting key c9f8018f9147f284a8e90d560971dbee
Sep 26 14:55:19 dist-test-slave-s4p1 krb5kdc[22589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:19.243173 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.235201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:54407 (local address 127.5.15.190:44343)
0926 14:55:19.235249 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.235251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.235434 (+ 183us) server_negotiation.cc:407] Connection header received
0926 14:55:19.236007 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.236009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.236023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.236104 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:19.237276 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.237764 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.238718 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.238970 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.239023 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.240911 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:19.240912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:19.240913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:19.240919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:19.242073 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:19.242417 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:19.242418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:19.242418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:19.242467 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:19.242663 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:19.242663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:19.242663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:19.242805 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:19.242863 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.243054 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.243111 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:55:19.243423 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.235138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:44343 (local address 127.5.15.131:54407)
0926 14:55:19.235297 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.235338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.235889 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.236181 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.236202 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.236977 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:19.237200 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.237205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.237839 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.237840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.238646 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.238649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.239037 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.239038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.239132 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.239542 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:19.239549 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:19.240835 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:19.242143 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:19.242154 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:19.242155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:19.242354 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:19.242525 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:19.242526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:19.242526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:19.242614 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:19.242853 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:19.242857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:19.242961 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.243040 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.243167 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260926 14:55:19.243921 23093 heartbeater.cc:347] Connected to a master server at 127.5.15.190:44343
I20260926 14:55:19.244046 23093 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:19.244249 23093 heartbeater.cc:511] Master 127.5.15.190:44343 requested a full tablet report, sending...
I20260926 14:55:19.244611 22633 ts_manager.cc:194] Registered new tserver with Master: 72f08b704f454cde971bb8437217609e (127.5.15.131:46845)
I20260926 14:55:19.244989 22633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:54407
I20260926 14:55:19.247795 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:55:19.255319 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.249119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.5.15.190:44343)
0926 14:55:19.249173 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.249175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.249290 (+ 115us) server_negotiation.cc:407] Connection header received
0926 14:55:19.249324 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.249325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.249332 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.249386 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:19.249904 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.250335 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.251111 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.251279 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.251329 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.252919 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:19.252921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:19.252921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:19.252928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:19.254174 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:19.254586 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:19.254587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:19.254587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:19.254636 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:19.254889 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:19.254889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:19.254890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:19.255010 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:19.255093 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.255216 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.255263 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:55:19.256714 22633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:55:19.258240 22633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:55:19.264662 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.260758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.5.15.130:43767)
0926 14:55:19.260908 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.260912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.260924 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:55:19.260962 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.260964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.260975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.261071 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:19.261299 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.262423 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.263702 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.264253 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.264348 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.264421 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.264539 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.264578 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260926 14:55:19.264662 23101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.260405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46845 (local address 127.0.0.1:52086)
0926 14:55:19.260505 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.260541 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.260622 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.261126 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.261129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.261132 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:19.261209 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.261211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.262200 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.262201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.263445 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.263448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.264363 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.264364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.264437 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.264446 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.264493 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.264553 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260926 14:55:19.264675 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.260639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36935 (local address 127.0.0.1:59386)
0926 14:55:19.260763 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.260773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.260812 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.261269 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.261270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.261272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:19.261303 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.261304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.262590 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.262590 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.263461 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.263463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.264361 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.264363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.264438 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.264447 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.264493 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.264605 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":92,"thread_start_us":86,"threads_started":1}
I20260926 14:55:19.264813 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.260509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:43767 (local address 127.0.0.1:51578)
0926 14:55:19.260634 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.260645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.260696 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.261165 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.261166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.261168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:19.261219 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.261221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.262525 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.262526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.263607 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.263609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.264347 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.264348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.264433 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.264446 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.264519 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.264565 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260926 14:55:19.265038 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.260506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.5.15.131:46845)
0926 14:55:19.260943 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.260948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.260960 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:55:19.261000 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.261003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.261018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.261320 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:19.261407 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.262088 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.263537 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.264255 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.264787 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.264859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.264887 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.264938 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":81,"threads_started":1}
I20260926 14:55:19.265434 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.260912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.5.15.129:36935)
0926 14:55:19.261090 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.261094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.261110 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:55:19.261147 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.261149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.261163 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.261375 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:19.261444 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.262479 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.263576 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.264268 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.264541 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.264603 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.264628 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.264671 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260926 14:55:19.265609 23027 tablet_service.cc:1467] Processing CreateTablet for tablet 40946aca529b4101acda8937c110aab1 (DEFAULT_TABLE table=test-table [id=d0202d40e87d451d84bf63665b2b6aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:19.265638 22893 tablet_service.cc:1467] Processing CreateTablet for tablet 40946aca529b4101acda8937c110aab1 (DEFAULT_TABLE table=test-table [id=d0202d40e87d451d84bf63665b2b6aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:19.265956 23027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40946aca529b4101acda8937c110aab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:19.265959 22893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40946aca529b4101acda8937c110aab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:19.266167 22759 tablet_service.cc:1467] Processing CreateTablet for tablet 40946aca529b4101acda8937c110aab1 (DEFAULT_TABLE table=test-table [id=d0202d40e87d451d84bf63665b2b6aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:19.266470 22759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40946aca529b4101acda8937c110aab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:19.268687 23107 tablet_bootstrap.cc:492] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Bootstrap starting.
I20260926 14:55:19.269304 23108 tablet_bootstrap.cc:492] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Bootstrap starting.
I20260926 14:55:19.269361 23107 tablet_bootstrap.cc:654] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:19.270053 23108 tablet_bootstrap.cc:654] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:19.270095 23109 tablet_bootstrap.cc:492] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: Bootstrap starting.
I20260926 14:55:19.270134 23107 log.cc:824] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:19.270619 23108 log.cc:824] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:19.270828 23109 tablet_bootstrap.cc:654] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:19.271028 23107 tablet_bootstrap.cc:492] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: No bootstrap required, opened a new log
I20260926 14:55:19.271108 23107 ts_tablet_manager.cc:1397] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:19.271363 23109 log.cc:824] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:19.271423 23108 tablet_bootstrap.cc:492] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: No bootstrap required, opened a new log
I20260926 14:55:19.271485 23108 ts_tablet_manager.cc:1397] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:19.272009 23109 tablet_bootstrap.cc:492] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: No bootstrap required, opened a new log
I20260926 14:55:19.272090 23109 ts_tablet_manager.cc:1397] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 14:55:19.273098 23107 raft_consensus.cc:348] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.273284 23107 raft_consensus.cc:374] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:19.273320 23107 raft_consensus.cc:729] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f688fc4162544eb19cbd9a23776ca85f, State: Initialized, Role: FOLLOWER
I20260926 14:55:19.273346 23108 raft_consensus.cc:348] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.273449 23107 consensus_queue.cc:260] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.273492 23108 raft_consensus.cc:374] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:19.273523 23108 raft_consensus.cc:729] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f08b704f454cde971bb8437217609e, State: Initialized, Role: FOLLOWER
I20260926 14:55:19.273653 23108 consensus_queue.cc:260] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.273710 23107 ts_tablet_manager.cc:1428] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:19.273834 22959 heartbeater.cc:503] Master 127.5.15.190:44343 was elected leader, sending a full tablet report...
I20260926 14:55:19.273911 23108 ts_tablet_manager.cc:1428] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 14:55:19.274025 23093 heartbeater.cc:503] Master 127.5.15.190:44343 was elected leader, sending a full tablet report...
I20260926 14:55:19.274152 23109 raft_consensus.cc:348] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.274323 23109 raft_consensus.cc:374] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:19.274369 23109 raft_consensus.cc:729] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ba639f446a45258f6b8a30680ee738, State: Initialized, Role: FOLLOWER
I20260926 14:55:19.274489 23109 consensus_queue.cc:260] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.274806 23109 ts_tablet_manager.cc:1428] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 14:55:19.274813 22825 heartbeater.cc:503] Master 127.5.15.190:44343 was elected leader, sending a full tablet report...
W20260926 14:55:19.345582 22960 tablet.cc:2369] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:55:19.430966 23113 raft_consensus.cc:482] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:19.431141 23113 raft_consensus.cc:504] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.431509 23113 leader_election.cc:288] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72f08b704f454cde971bb8437217609e (127.5.15.131:46845), 07ba639f446a45258f6b8a30680ee738 (127.5.15.129:36935)
I20260926 14:55:19.435210 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.431919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:48967 (local address 127.5.15.129:36935)
0926 14:55:19.431969 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.431971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.432099 (+ 128us) server_negotiation.cc:407] Connection header received
0926 14:55:19.432164 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.432165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.432178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.432232 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:19.432364 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.433067 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.434307 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.434868 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.434942 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.434950 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.435107 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.435144 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:55:19.435220 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.431804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:59165 (local address 127.5.15.131:46845)
0926 14:55:19.431876 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.431878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.431888 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:19.431916 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.431917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.431926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.431980 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:19.432154 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.432939 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.434314 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.434871 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.434942 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.434950 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.435111 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.435143 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:55:19.435230 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.431716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:46845 (local address 127.5.15.130:59165)
0926 14:55:19.431772 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.431784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.431857 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.431990 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.431991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.431993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:19.432073 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.432075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.433024 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.433025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.434215 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.434218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.434951 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.434952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.435031 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.435032 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.435088 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.435138 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":7}
I20260926 14:55:19.435237 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.431847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36935 (local address 127.5.15.130:48967)
0926 14:55:19.432023 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.432033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.432094 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.432246 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.432247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.432248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:19.432301 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.432302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.433159 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.433159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.434217 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.434218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.434951 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.434952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.435027 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.435031 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.435088 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.435137 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
I20260926 14:55:19.435560 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40946aca529b4101acda8937c110aab1" candidate_uuid: "f688fc4162544eb19cbd9a23776ca85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f08b704f454cde971bb8437217609e" is_pre_election: true
I20260926 14:55:19.435616 22779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40946aca529b4101acda8937c110aab1" candidate_uuid: "f688fc4162544eb19cbd9a23776ca85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ba639f446a45258f6b8a30680ee738" is_pre_election: true
I20260926 14:55:19.435714 23047 raft_consensus.cc:2454] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f688fc4162544eb19cbd9a23776ca85f in term 0.
I20260926 14:55:19.435799 22779 raft_consensus.cc:2454] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f688fc4162544eb19cbd9a23776ca85f in term 0.
I20260926 14:55:19.435921 22846 leader_election.cc:302] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72f08b704f454cde971bb8437217609e, f688fc4162544eb19cbd9a23776ca85f; no voters:
I20260926 14:55:19.436079 23113 raft_consensus.cc:2781] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:55:19.436147 23113 raft_consensus.cc:482] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:55:19.436172 23113 raft_consensus.cc:3037] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:19.436894 23113 raft_consensus.cc:504] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.437022 23113 leader_election.cc:288] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [CANDIDATE]: Term 1 election: Requested vote from peers 72f08b704f454cde971bb8437217609e (127.5.15.131:46845), 07ba639f446a45258f6b8a30680ee738 (127.5.15.129:36935)
I20260926 14:55:19.437247 23047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40946aca529b4101acda8937c110aab1" candidate_uuid: "f688fc4162544eb19cbd9a23776ca85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72f08b704f454cde971bb8437217609e"
I20260926 14:55:19.437278 22779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40946aca529b4101acda8937c110aab1" candidate_uuid: "f688fc4162544eb19cbd9a23776ca85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07ba639f446a45258f6b8a30680ee738"
I20260926 14:55:19.437327 23047 raft_consensus.cc:3037] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:19.437338 22779 raft_consensus.cc:3037] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:19.438233 23047 raft_consensus.cc:2454] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f688fc4162544eb19cbd9a23776ca85f in term 1.
I20260926 14:55:19.438241 22779 raft_consensus.cc:2454] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f688fc4162544eb19cbd9a23776ca85f in term 1.
I20260926 14:55:19.438401 22846 leader_election.cc:302] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07ba639f446a45258f6b8a30680ee738, f688fc4162544eb19cbd9a23776ca85f; no voters:
I20260926 14:55:19.438500 23113 raft_consensus.cc:2781] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:19.438632 23113 raft_consensus.cc:686] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 1 LEADER]: Becoming Leader. State: Replica: f688fc4162544eb19cbd9a23776ca85f, State: Running, Role: LEADER
I20260926 14:55:19.438882 23113 consensus_queue.cc:237] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } }
I20260926 14:55:19.439703 22633 catalog_manager.cc:5495] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f reported cstate change: term changed from 0 to 1, leader changed from <none> to f688fc4162544eb19cbd9a23776ca85f (127.5.15.130). New cstate: current_term: 1 leader_uuid: "f688fc4162544eb19cbd9a23776ca85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f688fc4162544eb19cbd9a23776ca85f" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 43767 } health_report { overall_health: HEALTHY } } }
W20260926 14:55:19.441224 22826 tablet.cc:2369] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:55:19.445564 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.443216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.5.15.130:43767)
0926 14:55:19.443258 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.443260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.443269 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:19.443343 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.443343 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.443368 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.443453 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:19.443588 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.444029 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.444769 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.444982 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.445053 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.445209 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:19.445303 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:19.445426 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.445458 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.445467 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:55:19.447114 22779 raft_consensus.cc:1261] T 40946aca529b4101acda8937c110aab1 P 07ba639f446a45258f6b8a30680ee738 [term 1 FOLLOWER]: Refusing update from remote peer f688fc4162544eb19cbd9a23776ca85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:19.447129 23047 raft_consensus.cc:1261] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 1 FOLLOWER]: Refusing update from remote peer f688fc4162544eb19cbd9a23776ca85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:19.447409 23113 consensus_queue.cc:1035] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "72f08b704f454cde971bb8437217609e" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 46845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:19.447568 23117 consensus_queue.cc:1035] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ba639f446a45258f6b8a30680ee738" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:19.450083 23121 mvcc.cc:205] Tried to move back new op lower bound from 7333619791652913152 to 7333619791622144000. Current Snapshot: MvccSnapshot[applied={T|T < 7333619791652913152}]
I20260926 14:55:19.450766 23122 mvcc.cc:205] Tried to move back new op lower bound from 7333619791652913152 to 7333619791622144000. Current Snapshot: MvccSnapshot[applied={T|T < 7333619791652913152 or (T in {7333619791652913152})}]
I20260926 14:55:19.452000 22634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:19.452090 22634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:19.452832 22634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0758f649d164f7687bebd65546715a1: Sending DeleteTablet for 3 replicas of tablet 40946aca529b4101acda8937c110aab1
I20260926 14:55:19.453136 22893 tablet_service.cc:1514] Processing DeleteTablet for tablet 40946aca529b4101acda8937c110aab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:19 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:51578
I20260926 14:55:19.453147 23027 tablet_service.cc:1514] Processing DeleteTablet for tablet 40946aca529b4101acda8937c110aab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:19 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:52086
I20260926 14:55:19.453167 22759 tablet_service.cc:1514] Processing DeleteTablet for tablet 40946aca529b4101acda8937c110aab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:19 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:59386
I20260926 14:55:19.453347 23130 tablet_replica.cc:330] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: stopping tablet replica
I20260926 14:55:19.453465 23130 raft_consensus.cc:2229] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:19.453570 23130 raft_consensus.cc:2258] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:19.453697 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22696
I20260926 14:55:19.453785 23131 tablet_replica.cc:330] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: stopping tablet replica
I20260926 14:55:19.453868 23131 raft_consensus.cc:2229] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:55:19.453964 23131 raft_consensus.cc:2258] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:19.454006 23130 ts_tablet_manager.cc:1905] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:19.455332 23131 ts_tablet_manager.cc:1905] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:19.455760 23130 ts_tablet_manager.cc:1918] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:19.455823 23130 log.cc:1196] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/40946aca529b4101acda8937c110aab1
I20260926 14:55:19.456097 23130 ts_tablet_manager.cc:1939] T 40946aca529b4101acda8937c110aab1 P 72f08b704f454cde971bb8437217609e: Deleting consensus metadata
I20260926 14:55:19.456553 22618 catalog_manager.cc:4841] TS 72f08b704f454cde971bb8437217609e (127.5.15.131:46845): tablet 40946aca529b4101acda8937c110aab1 (table test-table [id=d0202d40e87d451d84bf63665b2b6aa0]) successfully deleted
I20260926 14:55:19.456985 23131 ts_tablet_manager.cc:1918] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:19.457077 23131 log.cc:1196] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/40946aca529b4101acda8937c110aab1
I20260926 14:55:19.457314 23131 ts_tablet_manager.cc:1939] T 40946aca529b4101acda8937c110aab1 P f688fc4162544eb19cbd9a23776ca85f: Deleting consensus metadata
I20260926 14:55:19.457711 22620 catalog_manager.cc:4841] TS f688fc4162544eb19cbd9a23776ca85f (127.5.15.130:43767): tablet 40946aca529b4101acda8937c110aab1 (table test-table [id=d0202d40e87d451d84bf63665b2b6aa0]) successfully deleted
W20260926 14:55:19.460466 22618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:36935 (error 108)
I20260926 14:55:19.460752 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22830
I20260926 14:55:19.461369 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.460825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36935 (local address 127.0.0.1:59390)
0926 14:55:19.461294 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.461336 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:36935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447}
W20260926 14:55:19.461525 22618 catalog_manager.cc:4584] TS 07ba639f446a45258f6b8a30680ee738 (127.5.15.129:36935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40946aca529b4101acda8937c110aab1: Network error: Client connection negotiation failed: client connection to 127.5.15.129:36935: connect: Connection refused (error 111)
I20260926 14:55:19.466300 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22964
I20260926 14:55:19.470539 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 22602
2026-09-26T14:55:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1948 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T14:55:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:19Z Disabled control of system clock
I20260926 14:55:19.497531 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:46263
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:38475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:46263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:19.575532 23136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:19.575778 23136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:19.575812 23136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:19.576946 23136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:19.577010 23136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:19.577033 23136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:19.577067 23136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:19.577095 23136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:19.578439 23136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:46263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:19.578714 23136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:19.579072 23136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:19.583940 23142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.584152 23144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.584162 23141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.584575 23136 server_base.cc:1034] running on GCE node
I20260926 14:55:19.584784 23136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:19.585089 23136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:19.586254 23136 hybrid_clock.cc:648] HybridClock initialized: now 1790434519586231 us; error 34 us; skew 500 ppm
I20260926 14:55:19.592235 23136 webserver.cc:460] Webserver started at http://127.5.15.190:37125/ using document root <none> and password file <none>
I20260926 14:55:19.592490 23136 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:19.592561 23136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:19.592716 23136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:19.593775 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "1e1138d2081649f3a8f3753a2bf3e336"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "f2b1134b686e7fbb30697ca769f5d458"
server_key_iv: "6e79a912a6da3f2e71b4fe3957520407"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.594151 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "1e1138d2081649f3a8f3753a2bf3e336"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "f2b1134b686e7fbb30697ca769f5d458"
server_key_iv: "6e79a912a6da3f2e71b4fe3957520407"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.595491 23136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:19.596243 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.596402 23136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.596503 23136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "1e1138d2081649f3a8f3753a2bf3e336"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "f2b1134b686e7fbb30697ca769f5d458"
server_key_iv: "6e79a912a6da3f2e71b4fe3957520407"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.596572 23136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:19.608973 23136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:19.609246 23136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:19.609391 23136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:19.613471 23136 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:46263
I20260926 14:55:19.613498 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:46263 every 8 connection(s)
I20260926 14:55:19.613819 23136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:19.614709 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:19.616837 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336: Bootstrap starting.
I20260926 14:55:19.617393 23203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:19.617674 23203 log.cc:824] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:19.618320 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336: No bootstrap required, opened a new log
I20260926 14:55:19.619889 23203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1138d2081649f3a8f3753a2bf3e336" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46263 } }
I20260926 14:55:19.620023 23203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:19.620043 23203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e1138d2081649f3a8f3753a2bf3e336, State: Initialized, Role: FOLLOWER
I20260926 14:55:19.620131 23203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1138d2081649f3a8f3753a2bf3e336" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46263 } }
I20260926 14:55:19.620191 23203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:19.620226 23203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:19.620266 23203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:19.620718 23203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1138d2081649f3a8f3753a2bf3e336" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46263 } }
I20260926 14:55:19.620822 23203 leader_election.cc:302] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e1138d2081649f3a8f3753a2bf3e336; no voters:
I20260926 14:55:19.620996 23203 leader_election.cc:288] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:19.621084 23206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:19.621254 23203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:19.621264 23206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [term 1 LEADER]: Becoming Leader. State: Replica: 1e1138d2081649f3a8f3753a2bf3e336, State: Running, Role: LEADER
I20260926 14:55:19.621388 23206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1138d2081649f3a8f3753a2bf3e336" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46263 } }
I20260926 14:55:19.621841 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23136
I20260926 14:55:19.621935 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:19.621894 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e1138d2081649f3a8f3753a2bf3e336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1138d2081649f3a8f3753a2bf3e336" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46263 } } }
I20260926 14:55:19.621973 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:19.622128 5182 external_mini_cluster.cc:1428] Setting key d89b3961424455911a43568d43dffe72
I20260926 14:55:19.622144 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e1138d2081649f3a8f3753a2bf3e336. Latest consensus state: current_term: 1 leader_uuid: "1e1138d2081649f3a8f3753a2bf3e336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e1138d2081649f3a8f3753a2bf3e336" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 46263 } } }
I20260926 14:55:19.622176 23215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:19.622200 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:19.622650 23215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:19.624255 23215 catalog_manager.cc:1347] Generated new cluster ID: 79d945ef1c1544fd94cda84801c5ec9e
I20260926 14:55:19.624310 23215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:19.628407 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.622863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.5.15.190:46263)
0926 14:55:19.623029 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.623037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.623271 (+ 234us) server_negotiation.cc:407] Connection header received
0926 14:55:19.623392 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.623400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.623465 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.623604 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:19.623840 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.625237 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.626333 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.626945 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.627052 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.627755 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:19.627759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:19.627759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:19.627952 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:19.627991 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:19.628004 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:19.628065 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.628147 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.628215 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260926 14:55:19.630237 23215 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:19.630896 23215 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:19.640641 23215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336: Generated new TSK 0
I20260926 14:55:19.640808 23215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:19.644810 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46263
--builtin_ntp_servers=127.5.15.148:38475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:19.724303 23227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:19.724517 23227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:19.724555 23227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:19.725667 23227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:19.725730 23227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:19.725773 23227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:55:19.727173 23227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:19.727421 23227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:19.727701 23227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:19.733328 23233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.733343 23235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.733359 23232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.733865 23227 server_base.cc:1034] running on GCE node
I20260926 14:55:19.734050 23227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:19.734279 23227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:19.735430 23227 hybrid_clock.cc:648] HybridClock initialized: now 1790434519735413 us; error 29 us; skew 500 ppm
I20260926 14:55:19.736752 23227 webserver.cc:460] Webserver started at http://127.5.15.129:41953/ using document root <none> and password file <none>
I20260926 14:55:19.737007 23227 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:19.737063 23227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:19.737181 23227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:19.738481 23227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "c3c45443dbce44e095ef84f87282bbb9"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "eff81008e241f7ca26d1549ec2a9f96e"
server_key_iv: "d7be81bd682c9244f6bf8627c6bb6d9e"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.738878 23227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "c3c45443dbce44e095ef84f87282bbb9"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "eff81008e241f7ca26d1549ec2a9f96e"
server_key_iv: "d7be81bd682c9244f6bf8627c6bb6d9e"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.740391 23227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:19.741189 23241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.741375 23227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:19.741462 23227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "c3c45443dbce44e095ef84f87282bbb9"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "eff81008e241f7ca26d1549ec2a9f96e"
server_key_iv: "d7be81bd682c9244f6bf8627c6bb6d9e"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.741532 23227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:19.761813 23227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:19.762053 23227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:19.762146 23227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:19.762313 23227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:19.762629 23227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:19.762662 23227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.762718 23227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:19.762763 23227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.768409 23227 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:46243
I20260926 14:55:19.768461 23354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:46243 every 8 connection(s)
I20260926 14:55:19.768775 23227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:55:19.769673 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23227
I20260926 14:55:19.769771 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:55:19.769971 5182 external_mini_cluster.cc:1428] Setting key c5d23a22c86bdde00cfb7eb4e883d344
I20260926 14:55:19.771013 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46263
--builtin_ntp_servers=127.5.15.148:38475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:55:19.774416 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.769424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:60499 (local address 127.5.15.190:46263)
0926 14:55:19.769465 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.769468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.769726 (+ 258us) server_negotiation.cc:407] Connection header received
0926 14:55:19.770407 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.770408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.770418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.770457 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:19.771363 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.771830 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.772950 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.773150 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.773229 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.774139 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:19.774141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:19.774141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:19.774146 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:19.774165 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:19.774169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:19.774227 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.774307 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.774336 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:55:19.774655 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.769405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:46263 (local address 127.5.15.129:60499)
0926 14:55:19.769587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.769636 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.770296 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.770526 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.770529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.770539 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:19.771277 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.771284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.771924 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.771925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.772847 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.772850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.773239 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.773240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.773383 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.773992 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:19.773992 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:19.774027 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:19.774057 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:19.774228 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:19.774234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:19.774237 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.774308 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.774404 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260926 14:55:19.775162 23355 heartbeater.cc:347] Connected to a master server at 127.5.15.190:46263
I20260926 14:55:19.775261 23355 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:19.775434 23355 heartbeater.cc:511] Master 127.5.15.190:46263 requested a full tablet report, sending...
I20260926 14:55:19.775897 23167 ts_manager.cc:194] Registered new tserver with Master: c3c45443dbce44e095ef84f87282bbb9 (127.5.15.129:46243)
I20260926 14:55:19.776491 23167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.129:60499
W20260926 14:55:19.849881 23358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:19.850111 23358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:19.850147 23358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:19.851246 23358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:19.851305 23358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:19.851344 23358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:55:19.852635 23358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:19.852900 23358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:19.853189 23358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:19.858172 23363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.858220 23358 server_base.cc:1034] running on GCE node
W20260926 14:55:19.858219 23364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.858179 23366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.858610 23358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:19.858891 23358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:19.860029 23358 hybrid_clock.cc:648] HybridClock initialized: now 1790434519860020 us; error 29 us; skew 500 ppm
I20260926 14:55:19.861235 23358 webserver.cc:460] Webserver started at http://127.5.15.130:36063/ using document root <none> and password file <none>
I20260926 14:55:19.861476 23358 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:19.861546 23358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:19.861681 23358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:19.862668 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "6b8e245c40f44ae5a5242661a5073d90"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "f35882713ef645449db3dd81b028bc23"
server_key_iv: "fe00214b1f5fa1d772d19ba9ed3b628c"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.863101 23358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "6b8e245c40f44ae5a5242661a5073d90"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "f35882713ef645449db3dd81b028bc23"
server_key_iv: "fe00214b1f5fa1d772d19ba9ed3b628c"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.864344 23358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:19.865010 23372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.865190 23358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:19.865295 23358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "6b8e245c40f44ae5a5242661a5073d90"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "f35882713ef645449db3dd81b028bc23"
server_key_iv: "fe00214b1f5fa1d772d19ba9ed3b628c"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.865376 23358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:19.884759 23358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:19.885037 23358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:19.885178 23358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:19.885390 23358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:19.885717 23358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:19.885748 23358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.885769 23358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:19.885782 23358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.891283 23358 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:39447
I20260926 14:55:19.891331 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:39447 every 8 connection(s)
I20260926 14:55:19.891613 23358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:55:19.895702 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23358
I20260926 14:55:19.895798 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:55:19.896000 5182 external_mini_cluster.cc:1428] Setting key d972a85b14dc6f6eb799f7ab9a029609
I20260926 14:55:19.896778 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.892299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:49843 (local address 127.5.15.190:46263)
0926 14:55:19.892363 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:19.892365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:19.892595 (+ 230us) server_negotiation.cc:407] Connection header received
0926 14:55:19.893129 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:19.893130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:19.893146 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:19.893199 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:19.893844 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.894319 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.895268 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.895479 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.895555 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.896468 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:19.896468 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:19.896469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:19.896472 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:19.896487 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:19.896491 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:19.896531 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:19.896647 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 14:55:19.896676 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:55:19.896970 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:19.892284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:46263 (local address 127.5.15.130:49843)
0926 14:55:19.892481 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:19.892512 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:19.893008 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:19.893264 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:19.893268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:19.893277 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:19.893753 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.893756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.894423 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.894424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.895182 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:19.895184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:19.895561 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:19.895562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:19.895692 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:19.896272 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:19.896273 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:19.896310 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:19.896361 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:19.896543 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:19.896560 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:19.896562 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:55:19.896646 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:55:19.896750 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260926 14:55:19.897053 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46263
--builtin_ntp_servers=127.5.15.148:38475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:55:19.898522 23486 heartbeater.cc:347] Connected to a master server at 127.5.15.190:46263
I20260926 14:55:19.898622 23486 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:19.898846 23486 heartbeater.cc:511] Master 127.5.15.190:46263 requested a full tablet report, sending...
I20260926 14:55:19.899201 23167 ts_manager.cc:194] Registered new tserver with Master: 6b8e245c40f44ae5a5242661a5073d90 (127.5.15.130:39447)
I20260926 14:55:19.899619 23167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.130:49843
W20260926 14:55:19.976222 23489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:19.976432 23489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:19.976464 23489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:19.977538 23489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:19.977596 23489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:19.977645 23489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:55:19.979058 23489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:19.979302 23489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:19.979575 23489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:19.984315 23495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.984316 23497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:19.984337 23494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:19.984624 23489 server_base.cc:1034] running on GCE node
I20260926 14:55:19.984843 23489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:19.985096 23489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:19.986269 23489 hybrid_clock.cc:648] HybridClock initialized: now 1790434519986253 us; error 25 us; skew 500 ppm
I20260926 14:55:19.987629 23489 webserver.cc:460] Webserver started at http://127.5.15.131:37867/ using document root <none> and password file <none>
I20260926 14:55:19.987869 23489 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:19.987939 23489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:19.988086 23489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:19.989096 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "ebdbd40752324c34869730e6fb890405"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "ac0e6ff1f406829f4540f04308a3ffd7"
server_key_iv: "9def2ccb576de0e6c51ce1fe373e48e8"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.989451 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "ebdbd40752324c34869730e6fb890405"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "ac0e6ff1f406829f4540f04308a3ffd7"
server_key_iv: "9def2ccb576de0e6c51ce1fe373e48e8"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.990859 23489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:19.991727 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:19.991935 23489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:19.992025 23489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "ebdbd40752324c34869730e6fb890405"
format_stamp: "Formatted at 2026-09-26 14:55:19 on dist-test-slave-s4p1"
server_key: "ac0e6ff1f406829f4540f04308a3ffd7"
server_key_iv: "9def2ccb576de0e6c51ce1fe373e48e8"
server_key_version: "encryptionkey@0"
I20260926 14:55:19.992108 23489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:20.008250 23489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:20.008493 23489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:20.008628 23489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:20.008853 23489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:20.009184 23489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:20.009238 23489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:20.009268 23489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:20.009294 23489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:20.014981 23489 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:42279
I20260926 14:55:20.015015 23616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:42279 every 8 connection(s)
I20260926 14:55:20.015316 23489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:55:20.019986 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.016010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:47347 (local address 127.5.15.190:46263)
0926 14:55:20.016084 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.016086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.016253 (+ 167us) server_negotiation.cc:407] Connection header received
0926 14:55:20.016672 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.016673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.016683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.016728 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:20.017427 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.017913 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.018863 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.019043 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.019089 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.019762 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:20.019764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:20.019764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:20.019768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:20.019780 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:20.019783 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:20.019826 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.019908 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.019930 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:55:20.020164 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.015969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:46263 (local address 127.5.15.131:47347)
0926 14:55:20.016148 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.016177 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:20.016597 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:20.016784 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:20.016787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:20.016795 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:20.017246 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.017250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.017995 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.017996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.018786 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.018788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.019108 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.019109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.019207 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.019645 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:20.019645 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:20.019670 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:20.019689 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:20.019837 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:20.019841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:20.019842 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 14:55:20.019898 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:55:20.019969 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260926 14:55:20.020633 23617 heartbeater.cc:347] Connected to a master server at 127.5.15.190:46263
I20260926 14:55:20.020716 23617 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:20.020843 23617 heartbeater.cc:511] Master 127.5.15.190:46263 requested a full tablet report, sending...
I20260926 14:55:20.021170 23167 ts_manager.cc:194] Registered new tserver with Master: ebdbd40752324c34869730e6fb890405 (127.5.15.131:42279)
I20260926 14:55:20.021615 23167 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.131:47347
I20260926 14:55:20.022056 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23489
I20260926 14:55:20.022125 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:55:20.022316 5182 external_mini_cluster.cc:1428] Setting key 862445dbde2ca8b56f6ada692289d5fd
I20260926 14:55:20.023173 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:55:20.026950 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.024369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.5.15.190:46263)
0926 14:55:20.024420 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.024421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.024535 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:55:20.024568 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.024568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.024577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.024622 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:20.024740 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.025137 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.025796 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.025952 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.025996 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.026515 (+ 519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:20.026515 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:20.026516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:20.026519 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:20.026529 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:20.026532 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:20.026840 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.026897 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.026906 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:55:20.028509 23167 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:55:20.029923 23167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:55:20.035725 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.031895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:39447 (local address 127.0.0.1:50236)
0926 14:55:20.032048 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.032078 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:20.032136 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:20.032521 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:20.032522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:20.032523 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:20.032557 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.032558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.033615 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.033616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.034500 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.034502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.035437 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.035438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.035522 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.035533 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 14:55:20.035597 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:55:20.035640 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":49,"thread_start_us":88,"threads_started":1}
I20260926 14:55:20.035737 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.032213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50236 (local address 127.5.15.130:39447)
0926 14:55:20.032377 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.032381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.032391 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:20.032430 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.032432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.032447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.032526 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:20.032624 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.033538 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.034583 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.035340 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.035424 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.035517 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.035610 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.035644 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260926 14:55:20.035974 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.031896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:42279 (local address 127.0.0.1:40260)
0926 14:55:20.032179 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.032188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:20.032260 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:20.032424 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:20.032428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:20.032430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:20.032483 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.032484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.033599 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.033600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.034754 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.034773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.035760 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.035761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.035821 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.035824 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:20.035872 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:55:20.035906 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":24,"thread_start_us":60,"threads_started":1}
I20260926 14:55:20.036214 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.031955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.5.15.131:42279)
0926 14:55:20.032082 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.032086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.032250 (+ 164us) server_negotiation.cc:407] Connection header received
0926 14:55:20.032320 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.032322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.032334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.032406 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:20.032568 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.033506 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.034903 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.035577 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.035630 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.035728 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.036111 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.036146 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260926 14:55:20.036526 23420 tablet_service.cc:1467] Processing CreateTablet for tablet f1cbc545ce6c4bb2a30e68ffce07077f (DEFAULT_TABLE table=test-table [id=0de2734bdf9d4976afda446b6fb92c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:20.036692 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.032079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46243 (local address 127.0.0.1:46998)
0926 14:55:20.032702 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.032711 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:20.032758 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:20.033139 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:20.033140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:20.033141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:20.033180 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.033182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.034396 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.034397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.035393 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.035395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.036443 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.036444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.036505 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.036508 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:20.036596 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 14:55:20.036634 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":66,"threads_started":1}
I20260926 14:55:20.036816 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.032468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.5.15.129:46243)
0926 14:55:20.032643 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.032658 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.032860 (+ 202us) server_negotiation.cc:407] Connection header received
0926 14:55:20.032999 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.033001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.033016 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.033062 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:20.033258 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.034310 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.035499 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.036253 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.036327 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.036409 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.036613 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.036655 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260926 14:55:20.036754 23551 tablet_service.cc:1467] Processing CreateTablet for tablet f1cbc545ce6c4bb2a30e68ffce07077f (DEFAULT_TABLE table=test-table [id=0de2734bdf9d4976afda446b6fb92c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:20.036875 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cbc545ce6c4bb2a30e68ffce07077f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:20.037055 23551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cbc545ce6c4bb2a30e68ffce07077f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:20.037395 23289 tablet_service.cc:1467] Processing CreateTablet for tablet f1cbc545ce6c4bb2a30e68ffce07077f (DEFAULT_TABLE table=test-table [id=0de2734bdf9d4976afda446b6fb92c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:20.037616 23289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cbc545ce6c4bb2a30e68ffce07077f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:20.040316 23633 tablet_bootstrap.cc:492] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Bootstrap starting.
I20260926 14:55:20.040341 23631 tablet_bootstrap.cc:492] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9: Bootstrap starting.
I20260926 14:55:20.040987 23631 tablet_bootstrap.cc:654] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:20.040987 23633 tablet_bootstrap.cc:654] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:20.041319 23631 log.cc:824] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:20.041317 23633 log.cc:824] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:20.041638 23632 tablet_bootstrap.cc:492] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Bootstrap starting.
I20260926 14:55:20.042042 23633 tablet_bootstrap.cc:492] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: No bootstrap required, opened a new log
I20260926 14:55:20.042119 23633 ts_tablet_manager.cc:1397] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:20.042248 23632 tablet_bootstrap.cc:654] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:20.042387 23631 tablet_bootstrap.cc:492] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9: No bootstrap required, opened a new log
I20260926 14:55:20.042454 23631 ts_tablet_manager.cc:1397] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:20.042552 23632 log.cc:824] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:20.043560 23632 tablet_bootstrap.cc:492] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: No bootstrap required, opened a new log
I20260926 14:55:20.043619 23632 ts_tablet_manager.cc:1397] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:20.044402 23633 raft_consensus.cc:348] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.044553 23633 raft_consensus.cc:374] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:20.044595 23633 raft_consensus.cc:729] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8e245c40f44ae5a5242661a5073d90, State: Initialized, Role: FOLLOWER
I20260926 14:55:20.044704 23633 consensus_queue.cc:260] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.044955 23633 ts_tablet_manager.cc:1428] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:20.045094 23486 heartbeater.cc:503] Master 127.5.15.190:46263 was elected leader, sending a full tablet report...
I20260926 14:55:20.044997 23631 raft_consensus.cc:348] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.045162 23631 raft_consensus.cc:374] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:20.045207 23631 raft_consensus.cc:729] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c45443dbce44e095ef84f87282bbb9, State: Initialized, Role: FOLLOWER
I20260926 14:55:20.045367 23631 consensus_queue.cc:260] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.045591 23631 ts_tablet_manager.cc:1428] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:20.045653 23355 heartbeater.cc:503] Master 127.5.15.190:46263 was elected leader, sending a full tablet report...
I20260926 14:55:20.046046 23632 raft_consensus.cc:348] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.046180 23632 raft_consensus.cc:374] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:20.046222 23632 raft_consensus.cc:729] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdbd40752324c34869730e6fb890405, State: Initialized, Role: FOLLOWER
I20260926 14:55:20.046324 23632 consensus_queue.cc:260] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.046566 23632 ts_tablet_manager.cc:1428] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 14:55:20.046619 23617 heartbeater.cc:503] Master 127.5.15.190:46263 was elected leader, sending a full tablet report...
I20260926 14:55:20.123510 23637 raft_consensus.cc:482] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:20.123623 23637 raft_consensus.cc:504] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.124048 23637 leader_election.cc:288] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebdbd40752324c34869730e6fb890405 (127.5.15.131:42279), c3c45443dbce44e095ef84f87282bbb9 (127.5.15.129:46243)
I20260926 14:55:20.127895 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.124344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:52435 (local address 127.5.15.131:42279)
0926 14:55:20.124416 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.124418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.124428 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:20.124463 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.124464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.124475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.124533 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:20.124703 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.125454 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.127025 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.127558 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.127633 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.127641 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.127792 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.127827 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:55:20.127895 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.124487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:56079 (local address 127.5.15.129:46243)
0926 14:55:20.124530 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.124532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.124636 (+ 104us) server_negotiation.cc:407] Connection header received
0926 14:55:20.124672 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.124673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.124684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.124743 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:20.124829 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.125512 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.127026 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.127558 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.127633 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.127641 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.127792 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.127827 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:55:20.127928 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.124282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:42279 (local address 127.5.15.130:52435)
0926 14:55:20.124341 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.124354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:20.124403 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:20.124585 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:20.124586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:20.124588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:20.124652 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.124654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.125580 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.125581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.126923 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.126926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.127641 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.127642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.127715 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.127719 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:20.127776 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:55:20.127822 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":89}
I20260926 14:55:20.127933 23640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.124423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46243 (local address 127.5.15.130:56079)
0926 14:55:20.124565 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.124573 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:20.124625 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:20.124746 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:20.124747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:20.124748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:20.124788 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.124789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.125595 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.125595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.126928 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:20.126930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.127642 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.127643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:20.127719 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.127722 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:20.127776 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 14:55:20.127822 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":61,"thread_start_us":91,"threads_started":1}
I20260926 14:55:20.128315 23571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cbc545ce6c4bb2a30e68ffce07077f" candidate_uuid: "6b8e245c40f44ae5a5242661a5073d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdbd40752324c34869730e6fb890405" is_pre_election: true
I20260926 14:55:20.128391 23309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cbc545ce6c4bb2a30e68ffce07077f" candidate_uuid: "6b8e245c40f44ae5a5242661a5073d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3c45443dbce44e095ef84f87282bbb9" is_pre_election: true
I20260926 14:55:20.128494 23571 raft_consensus.cc:2454] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8e245c40f44ae5a5242661a5073d90 in term 0.
I20260926 14:55:20.128504 23309 raft_consensus.cc:2454] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8e245c40f44ae5a5242661a5073d90 in term 0.
I20260926 14:55:20.128732 23374 leader_election.cc:302] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8e245c40f44ae5a5242661a5073d90, c3c45443dbce44e095ef84f87282bbb9; no voters:
I20260926 14:55:20.128850 23637 raft_consensus.cc:2781] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:55:20.128906 23637 raft_consensus.cc:482] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:55:20.128930 23637 raft_consensus.cc:3037] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:20.129596 23637 raft_consensus.cc:504] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.129720 23637 leader_election.cc:288] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [CANDIDATE]: Term 1 election: Requested vote from peers ebdbd40752324c34869730e6fb890405 (127.5.15.131:42279), c3c45443dbce44e095ef84f87282bbb9 (127.5.15.129:46243)
I20260926 14:55:20.129873 23571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cbc545ce6c4bb2a30e68ffce07077f" candidate_uuid: "6b8e245c40f44ae5a5242661a5073d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdbd40752324c34869730e6fb890405"
I20260926 14:55:20.129876 23309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cbc545ce6c4bb2a30e68ffce07077f" candidate_uuid: "6b8e245c40f44ae5a5242661a5073d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3c45443dbce44e095ef84f87282bbb9"
I20260926 14:55:20.129945 23571 raft_consensus.cc:3037] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:20.129980 23309 raft_consensus.cc:3037] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:20.130627 23571 raft_consensus.cc:2454] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8e245c40f44ae5a5242661a5073d90 in term 1.
I20260926 14:55:20.130790 23374 leader_election.cc:302] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8e245c40f44ae5a5242661a5073d90, ebdbd40752324c34869730e6fb890405; no voters:
I20260926 14:55:20.130832 23309 raft_consensus.cc:2454] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8e245c40f44ae5a5242661a5073d90 in term 1.
I20260926 14:55:20.130868 23637 raft_consensus.cc:2781] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:20.131001 23637 raft_consensus.cc:686] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 1 LEADER]: Becoming Leader. State: Replica: 6b8e245c40f44ae5a5242661a5073d90, State: Running, Role: LEADER
I20260926 14:55:20.131088 23637 consensus_queue.cc:237] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } }
I20260926 14:55:20.131805 23167 catalog_manager.cc:5495] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b8e245c40f44ae5a5242661a5073d90 (127.5.15.130). New cstate: current_term: 1 leader_uuid: "6b8e245c40f44ae5a5242661a5073d90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b8e245c40f44ae5a5242661a5073d90" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 39447 } health_report { overall_health: HEALTHY } } }
W20260926 14:55:20.142086 23487 tablet.cc:2369] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:55:20.150300 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.148177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.5.15.130:39447)
0926 14:55:20.148226 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:20.148228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:20.148238 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:20.148315 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:20.148315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:20.148333 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:20.148396 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:20.148497 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.148950 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.149625 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:20.149799 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:20.149846 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:20.149970 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:20.150052 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:20.150121 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:20.150160 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 14:55:20.150167 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 14:55:20.152096 23309 raft_consensus.cc:1261] T f1cbc545ce6c4bb2a30e68ffce07077f P c3c45443dbce44e095ef84f87282bbb9 [term 1 FOLLOWER]: Refusing update from remote peer 6b8e245c40f44ae5a5242661a5073d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:20.152148 23571 raft_consensus.cc:1261] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 1 FOLLOWER]: Refusing update from remote peer 6b8e245c40f44ae5a5242661a5073d90: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:20.152458 23637 consensus_queue.cc:1035] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebdbd40752324c34869730e6fb890405" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:20.152613 23641 consensus_queue.cc:1035] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3c45443dbce44e095ef84f87282bbb9" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 46243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:20.155412 23645 mvcc.cc:205] Tried to move back new op lower bound from 7333619794540240896 to 7333619794457550848. Current Snapshot: MvccSnapshot[applied={T|T < 7333619794540240896}]
I20260926 14:55:20.157423 23166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:20.157523 23166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:20.158262 23166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e1138d2081649f3a8f3753a2bf3e336: Sending DeleteTablet for 3 replicas of tablet f1cbc545ce6c4bb2a30e68ffce07077f
I20260926 14:55:20.158622 23420 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cbc545ce6c4bb2a30e68ffce07077f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:20 UTC) from {username='slave'} at 127.0.0.1:50236
I20260926 14:55:20.158607 23551 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cbc545ce6c4bb2a30e68ffce07077f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:20 UTC) from {username='slave'} at 127.0.0.1:40260
I20260926 14:55:20.158645 23289 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cbc545ce6c4bb2a30e68ffce07077f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:20 UTC) from {username='slave'} at 127.0.0.1:46998
I20260926 14:55:20.158861 23654 tablet_replica.cc:330] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: stopping tablet replica
I20260926 14:55:20.158957 23654 raft_consensus.cc:2229] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:55:20.159071 23654 raft_consensus.cc:2258] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:20.159149 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23227
I20260926 14:55:20.159210 23655 tablet_replica.cc:330] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: stopping tablet replica
I20260926 14:55:20.159292 23655 raft_consensus.cc:2229] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:20.159376 23655 raft_consensus.cc:2258] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:20.159790 23654 ts_tablet_manager.cc:1905] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:20.161614 23654 ts_tablet_manager.cc:1918] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:20.161679 23654 log.cc:1196] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/f1cbc545ce6c4bb2a30e68ffce07077f
I20260926 14:55:20.161934 23654 ts_tablet_manager.cc:1939] T f1cbc545ce6c4bb2a30e68ffce07077f P 6b8e245c40f44ae5a5242661a5073d90: Deleting consensus metadata
I20260926 14:55:20.162402 23655 ts_tablet_manager.cc:1905] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:20.162448 23151 catalog_manager.cc:4841] TS 6b8e245c40f44ae5a5242661a5073d90 (127.5.15.130:39447): tablet f1cbc545ce6c4bb2a30e68ffce07077f (table test-table [id=0de2734bdf9d4976afda446b6fb92c91]) successfully deleted
I20260926 14:55:20.164026 23655 ts_tablet_manager.cc:1918] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:20.164100 23655 log.cc:1196] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/f1cbc545ce6c4bb2a30e68ffce07077f
I20260926 14:55:20.164350 23655 ts_tablet_manager.cc:1939] T f1cbc545ce6c4bb2a30e68ffce07077f P ebdbd40752324c34869730e6fb890405: Deleting consensus metadata
I20260926 14:55:20.164757 23152 catalog_manager.cc:4841] TS ebdbd40752324c34869730e6fb890405 (127.5.15.131:42279): tablet f1cbc545ce6c4bb2a30e68ffce07077f (table test-table [id=0de2734bdf9d4976afda446b6fb92c91]) successfully deleted
W20260926 14:55:20.165901 23152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:46243 (error 108)
I20260926 14:55:20.166095 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23358
I20260926 14:55:20.167124 23625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:20.166250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:46243 (local address 127.0.0.1:47004)
0926 14:55:20.167056 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:20.167098 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785}
W20260926 14:55:20.167315 23152 catalog_manager.cc:4584] TS c3c45443dbce44e095ef84f87282bbb9 (127.5.15.129:46243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1cbc545ce6c4bb2a30e68ffce07077f: Network error: Client connection negotiation failed: client connection to 127.5.15.129:46243: connect: Connection refused (error 111)
I20260926 14:55:20.171343 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23489
I20260926 14:55:20.176110 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23136
2026-09-26T14:55:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (704 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:20 dist-test-slave-s4p1 krb5kdc[23660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:20 dist-test-slave-s4p1 krb5kdc[23660](info): set up 2 sockets
Sep 26 14:55:20 dist-test-slave-s4p1 krb5kdc[23660](info): commencing operation
krb5kdc: starting...
W20260926 14:55:21.210422 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:21.321688 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:35613
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:21.399688 23673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:21.399883 23673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:21.399905 23673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:21.401026 23673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:21.401064 23673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:21.401078 23673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:21.401115 23673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:21.401144 23673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:21.402887 23673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:35613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:21.403169 23673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:21.403389 23673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:21.408018 23681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.408118 23679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.408047 23678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:21.408214 23673 server_base.cc:1034] running on GCE node
I20260926 14:55:21.408553 23673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:21.408855 23673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:21.410020 23673 hybrid_clock.cc:648] HybridClock initialized: now 1790434521410013 us; error 35 us; skew 500 ppm
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:21.412670 23673 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:21.413130 23673 webserver.cc:460] Webserver started at http://127.5.15.190:40159/ using document root <none> and password file <none>
I20260926 14:55:21.413364 23673 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:21.413429 23673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:21.413558 23673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:21.414510 23673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "362e07f4f10241a188e436b3090aa6a9"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "84dbf06f28b311c3c49a02f0d6829732"
server_key_iv: "234efb38e5dada93c9dce06381a6d4eb"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.414885 23673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "362e07f4f10241a188e436b3090aa6a9"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "84dbf06f28b311c3c49a02f0d6829732"
server_key_iv: "234efb38e5dada93c9dce06381a6d4eb"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.416163 23673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:21.416862 23688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.417032 23673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:21.417128 23673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "362e07f4f10241a188e436b3090aa6a9"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "84dbf06f28b311c3c49a02f0d6829732"
server_key_iv: "234efb38e5dada93c9dce06381a6d4eb"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.417230 23673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:21.433888 23673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:21.435828 23673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:21.435982 23673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:21.440052 23673 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:35613
I20260926 14:55:21.440089 23740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:35613 every 8 connection(s)
I20260926 14:55:21.440485 23673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:21.441290 23741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:21.443471 23741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9: Bootstrap starting.
I20260926 14:55:21.443980 23741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:21.444211 23741 log.cc:824] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:21.444820 23741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9: No bootstrap required, opened a new log
I20260926 14:55:21.445767 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23673
I20260926 14:55:21.445890 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:21.446069 5182 external_mini_cluster.cc:1428] Setting key aef1da4502993be9eeb028dafca8bd18
I20260926 14:55:21.446619 23741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362e07f4f10241a188e436b3090aa6a9" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35613 } }
I20260926 14:55:21.446846 23741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:21.446895 23741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362e07f4f10241a188e436b3090aa6a9, State: Initialized, Role: FOLLOWER
I20260926 14:55:21.447014 23741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362e07f4f10241a188e436b3090aa6a9" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35613 } }
I20260926 14:55:21.447089 23741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:21.447124 23741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:21.447178 23741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:21.448284 23741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362e07f4f10241a188e436b3090aa6a9" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35613 } }
I20260926 14:55:21.448371 23741 leader_election.cc:302] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 362e07f4f10241a188e436b3090aa6a9; no voters:
I20260926 14:55:21.448534 23741 leader_election.cc:288] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:21.448565 23746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:21.448669 23746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 362e07f4f10241a188e436b3090aa6a9, State: Running, Role: LEADER
I20260926 14:55:21.448778 23741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:21.448752 23746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362e07f4f10241a188e436b3090aa6a9" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35613 } }
I20260926 14:55:21.449154 23746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 362e07f4f10241a188e436b3090aa6a9. Latest consensus state: current_term: 1 leader_uuid: "362e07f4f10241a188e436b3090aa6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362e07f4f10241a188e436b3090aa6a9" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35613 } } }
I20260926 14:55:21.449226 23746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:21.449402 23748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "362e07f4f10241a188e436b3090aa6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "362e07f4f10241a188e436b3090aa6a9" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35613 } } }
I20260926 14:55:21.449461 23748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:21.450201 23750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:21.450675 23750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:21.452280 23750 catalog_manager.cc:1347] Generated new cluster ID: bfc4022891d84ef58df444f4f5def6ec
I20260926 14:55:21.452333 23750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:21.457465 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.446599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.5.15.190:35613)
0926 14:55:21.446820 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.446829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.446863 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:55:21.446974 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.446980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.447035 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.447176 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:21.448093 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.449733 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.450881 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.451464 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.451576 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.453969 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:21.453973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:21.453973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:21.453985 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:21.455686 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.456065 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.456068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.456068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.456167 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.456396 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.456397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.456397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.456624 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:21.457171 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.457227 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.457326 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260926 14:55:21.461038 23750 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:21.461581 23750 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:21.468191 23750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9: Generated new TSK 0
I20260926 14:55:21.468391 23750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.15.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
Entry for principal kudu/127.5.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab.
I20260926 14:55:21.496974 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:35613
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:21.575947 23767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:21.576165 23767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:21.576205 23767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:21.577342 23767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:21.577419 23767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:21.577462 23767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:55:21.578966 23767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.129.keytab
--principal=kudu/127.5.15.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:35613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:21.579211 23767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:21.579427 23767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:21.584152 23775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.584160 23773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.584200 23772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:21.584350 23767 server_base.cc:1034] running on GCE node
I20260926 14:55:21.584578 23767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:21.584796 23767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:21.585961 23767 hybrid_clock.cc:648] HybridClock initialized: now 1790434521585949 us; error 24 us; skew 500 ppm
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:21.588387 23767 init.cc:377] Logged in from keytab as kudu/127.5.15.129@KRBTEST.COM (short username kudu)
I20260926 14:55:21.588793 23767 webserver.cc:460] Webserver started at http://127.5.15.129:43985/ using document root <none> and password file <none>
I20260926 14:55:21.589004 23767 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:21.589069 23767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:21.589216 23767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:21.590216 23767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "6bd8d4f1cdfb4202863111465c624576"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "0a983a6fc8dc879b9ac2cad8f8e9d102"
server_key_iv: "a57dbaff0721dfe9e1421364b9a0c502"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.590574 23767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "6bd8d4f1cdfb4202863111465c624576"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "0a983a6fc8dc879b9ac2cad8f8e9d102"
server_key_iv: "a57dbaff0721dfe9e1421364b9a0c502"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.591917 23767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:21.592603 23782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.592785 23767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:21.592878 23767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "6bd8d4f1cdfb4202863111465c624576"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "0a983a6fc8dc879b9ac2cad8f8e9d102"
server_key_iv: "a57dbaff0721dfe9e1421364b9a0c502"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.592960 23767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:21.617137 23767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:21.619184 23767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:21.619365 23767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:21.619598 23767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:21.619951 23767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:21.620002 23767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.620042 23767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:21.620069 23767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.625697 23767 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:42089
I20260926 14:55:21.625742 23895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:42089 every 8 connection(s)
I20260926 14:55:21.626075 23767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.129@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:21.631723 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23767
I20260926 14:55:21.631801 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:55:21.631996 5182 external_mini_cluster.cc:1428] Setting key 20b21045e2f6adb1b0e8e0f2d2c3fb28
I20260926 14:55:21.634654 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.626796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:42445 (local address 127.5.15.190:35613)
0926 14:55:21.626853 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.626855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.627040 (+ 185us) server_negotiation.cc:407] Connection header received
0926 14:55:21.627492 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.627492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.627504 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.627557 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:21.628366 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.628837 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.629758 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.629943 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.629991 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.631937 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:21.631938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:21.631939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:21.631945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:21.633341 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.633738 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.633740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.633740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.633803 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.634047 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.634048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.634048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.634158 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:21.634231 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.634497 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.634576 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:55:21.634824 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.626747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:35613 (local address 127.5.15.129:42445)
0926 14:55:21.626923 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:21.626962 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:21.627416 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:21.627605 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:21.627608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:21.628117 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:21.628300 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.628304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.628905 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.628905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.629690 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.629692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.630011 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.630012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.630105 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.630490 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:21.630497 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:21.631853 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:21.633437 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:21.633444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:21.633446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:21.633651 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:21.633874 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:21.633875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:21.633875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:21.633977 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:21.634222 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:21.634228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:21.634373 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 14:55:21.634464 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 14:55:21.634598 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260926 14:55:21.635390 23896 heartbeater.cc:347] Connected to a master server at 127.5.15.190:35613
I20260926 14:55:21.635484 23896 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:21.635658 23896 heartbeater.cc:511] Master 127.5.15.190:35613 requested a full tablet report, sending...
I20260926 14:55:21.636144 23705 ts_manager.cc:194] Registered new tserver with Master: 6bd8d4f1cdfb4202863111465c624576 (127.5.15.129:42089)
I20260926 14:55:21.636737 23705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.129@KRBTEST.COM'} at 127.5.15.129:42445
WARNING: no policy specified for kudu/127.5.15.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
Entry for principal kudu/127.5.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab.
I20260926 14:55:21.655123 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:35613
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:21.733273 23901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:21.733475 23901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:21.733497 23901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:21.734618 23901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:21.734659 23901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:21.734723 23901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:55:21.736095 23901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.130.keytab
--principal=kudu/127.5.15.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:35613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:21.736352 23901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:21.736580 23901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:21.741245 23907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.741233 23909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.741359 23906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:21.741480 23901 server_base.cc:1034] running on GCE node
I20260926 14:55:21.741698 23901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:21.741990 23901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:21.743140 23901 hybrid_clock.cc:648] HybridClock initialized: now 1790434521743134 us; error 35 us; skew 500 ppm
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:21.745591 23901 init.cc:377] Logged in from keytab as kudu/127.5.15.130@KRBTEST.COM (short username kudu)
I20260926 14:55:21.746021 23901 webserver.cc:460] Webserver started at http://127.5.15.130:40425/ using document root <none> and password file <none>
I20260926 14:55:21.746239 23901 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:21.746302 23901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:21.746454 23901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:21.747438 23901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "46aff6972a0a4214af531a56c46149e8"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "30159e51e567a2c0c3ee436de15896a9"
server_key_iv: "1f1576c34af7b1fa9dea3e52c2543b0a"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.747805 23901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "46aff6972a0a4214af531a56c46149e8"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "30159e51e567a2c0c3ee436de15896a9"
server_key_iv: "1f1576c34af7b1fa9dea3e52c2543b0a"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.749083 23901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:21.749886 23916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.750042 23901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:21.750130 23901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "46aff6972a0a4214af531a56c46149e8"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "30159e51e567a2c0c3ee436de15896a9"
server_key_iv: "1f1576c34af7b1fa9dea3e52c2543b0a"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.750214 23901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:21.778517 23901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:21.780784 23901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:21.780944 23901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:21.781158 23901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:21.781448 23901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:21.781502 23901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.781540 23901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:21.781569 23901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.787068 23901 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:35187
I20260926 14:55:21.787101 24029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:35187 every 8 connection(s)
I20260926 14:55:21.787395 23901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:55:21.789088 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 23901
I20260926 14:55:21.789211 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:55:21.789395 5182 external_mini_cluster.cc:1428] Setting key 1a3fb47bcf4d88eae9c46947cb72bc83
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.130@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:21.796530 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.788125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:33519 (local address 127.5.15.190:35613)
0926 14:55:21.788195 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.788198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.788387 (+ 189us) server_negotiation.cc:407] Connection header received
0926 14:55:21.788798 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.788799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.788808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.788864 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:21.789967 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.790473 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.791440 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.791648 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.791736 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.793684 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:21.793685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:21.793686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:21.793692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:21.795118 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.795533 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.795534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.795535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.795616 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.795871 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.795872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.795872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.795981 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:21.796049 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.796350 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.796445 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:55:21.796715 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.788095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:35613 (local address 127.5.15.130:33519)
0926 14:55:21.788274 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:21.788307 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:21.788723 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:21.788905 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:21.788908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:21.789682 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:21.789872 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.789877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.790566 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.790567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.791349 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.791351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.791745 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.791745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.791871 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.792315 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:21.792322 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:21.793588 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:21.795216 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:21.795222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:21.795224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:21.795428 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:21.795695 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:21.795696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:21.795696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:21.795798 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:21.796052 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:21.796070 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:21.796220 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 14:55:21.796317 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 14:55:21.796472 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260926 14:55:21.797158 24030 heartbeater.cc:347] Connected to a master server at 127.5.15.190:35613
I20260926 14:55:21.797243 24030 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.15.131@KRBTEST.COM; defaulting to no policy
I20260926 14:55:21.797420 24030 heartbeater.cc:511] Master 127.5.15.190:35613 requested a full tablet report, sending...
I20260926 14:55:21.797788 23705 ts_manager.cc:194] Registered new tserver with Master: 46aff6972a0a4214af531a56c46149e8 (127.5.15.130:35187)
I20260926 14:55:21.798192 23705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.130@KRBTEST.COM'} at 127.5.15.130:33519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
Entry for principal kudu/127.5.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab.
I20260926 14:55:21.813115 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:35613
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:21.890591 24035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:21.890784 24035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:21.890806 24035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:21.891824 24035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:21.891855 24035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:21.891880 24035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:55:21.893211 24035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.131.keytab
--principal=kudu/127.5.15.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:35613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:21.893380 24035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:21.893533 24035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:21.898428 24040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.898505 24041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:21.898438 24043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:21.898687 24035 server_base.cc:1034] running on GCE node
I20260926 14:55:21.898904 24035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:21.899137 24035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:21.900283 24035 hybrid_clock.cc:648] HybridClock initialized: now 1790434521900273 us; error 28 us; skew 500 ppm
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:21.902539 24035 init.cc:377] Logged in from keytab as kudu/127.5.15.131@KRBTEST.COM (short username kudu)
I20260926 14:55:21.902933 24035 webserver.cc:460] Webserver started at http://127.5.15.131:36825/ using document root <none> and password file <none>
I20260926 14:55:21.903098 24035 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:21.903136 24035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:21.903223 24035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:21.904066 24035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "ecdb951ba7d940b38ea332dacecd85a1"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "8d48a7be60a1ad35690fa1d221584405"
server_key_iv: "1641e70324235108aaaaa5ee55ed5710"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.904356 24035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "ecdb951ba7d940b38ea332dacecd85a1"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "8d48a7be60a1ad35690fa1d221584405"
server_key_iv: "1641e70324235108aaaaa5ee55ed5710"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.905493 24035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:55:21.906150 24050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.906323 24035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:21.906420 24035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "ecdb951ba7d940b38ea332dacecd85a1"
format_stamp: "Formatted at 2026-09-26 14:55:21 on dist-test-slave-s4p1"
server_key: "8d48a7be60a1ad35690fa1d221584405"
server_key_iv: "1641e70324235108aaaaa5ee55ed5710"
server_key_version: "encryptionkey@0"
I20260926 14:55:21.906495 24035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:21.919847 24035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:21.921653 24035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:21.921798 24035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:21.921996 24035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:21.922287 24035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:21.922325 24035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.922344 24035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:21.922391 24035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:21.928261 24035 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:44089
I20260926 14:55:21.928295 24163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:44089 every 8 connection(s)
I20260926 14:55:21.928599 24035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:55:21 dist-test-slave-s4p1 krb5kdc[23660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.131@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:21.936650 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.929289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:49259 (local address 127.5.15.190:35613)
0926 14:55:21.929367 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.929370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.929544 (+ 174us) server_negotiation.cc:407] Connection header received
0926 14:55:21.929948 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.929949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.929958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.930008 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:21.930876 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.931359 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.932229 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.932402 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.932449 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.934322 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:21.934323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:21.934323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:21.934329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:21.935597 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.935936 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.935936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.935937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.935985 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.936178 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.936178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.936179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.936272 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:21.936320 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.936527 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.936583 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:55:21.936779 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.929262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:35613 (local address 127.5.15.131:49259)
0926 14:55:21.929428 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:21.929459 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:21.929871 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:21.930054 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:21.930057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:21.930579 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:21.930789 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.930796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.931432 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.931432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.932157 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.932159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.932467 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.932467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.932554 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.932932 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:21.932940 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:21.934242 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:21.935666 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:21.935671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:21.935672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:21.935874 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:21.936041 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:21.936041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:21.936041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:21.936128 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:21.936325 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:21.936328 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:21.936425 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 14:55:21.936498 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:55:21.936603 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 14:55:21.936969 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24035
I20260926 14:55:21.937074 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:55:21.937285 5182 external_mini_cluster.cc:1428] Setting key a7628d944a8b871f43258bf80b726e2f
I20260926 14:55:21.937264 24164 heartbeater.cc:347] Connected to a master server at 127.5.15.190:35613
I20260926 14:55:21.937378 24164 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:21.937551 24164 heartbeater.cc:511] Master 127.5.15.190:35613 requested a full tablet report, sending...
I20260926 14:55:21.937947 23705 ts_manager.cc:194] Registered new tserver with Master: ecdb951ba7d940b38ea332dacecd85a1 (127.5.15.131:44089)
I20260926 14:55:21.938409 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:55:21.938458 23705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.15.131@KRBTEST.COM'} at 127.5.15.131:49259
I20260926 14:55:21.945514 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.939893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.5.15.190:35613)
0926 14:55:21.939969 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.939971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.940062 (+ 91us) server_negotiation.cc:407] Connection header received
0926 14:55:21.940123 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.940124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.940134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.940188 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:21.940729 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.941168 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.941904 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.942102 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.942175 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.943243 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:21.943244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:21.943244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:21.943250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:21.944305 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.944656 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.944657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.944657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.944711 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:21.944940 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:21.944941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:21.944942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:21.945079 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:21.945181 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.945329 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.945420 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:55:21.946914 23705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:55:21.948441 23705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:55:21.954748 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.950675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42089 (local address 127.0.0.1:33124)
0926 14:55:21.951247 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:21.951256 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:21.951336 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:21.951566 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:21.951567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:21.951569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:21.951615 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.951617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.952468 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.952469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.953606 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.953608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.954438 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.954439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.954512 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.954524 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 14:55:21.954592 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:55:21.954642 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"spinlock_wait_cycles":960384,"thread_start_us":78,"threads_started":1}
I20260926 14:55:21.954743 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.950902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.5.15.129:42089)
0926 14:55:21.951070 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.951074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.951310 (+ 236us) server_negotiation.cc:407] Connection header received
0926 14:55:21.951410 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.951412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.951426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.951489 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:21.951695 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.952380 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.953673 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.954351 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.954424 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.954475 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.954613 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.954650 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260926 14:55:21.955175 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.950794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.5.15.131:44089)
0926 14:55:21.950933 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.950937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.951218 (+ 281us) server_negotiation.cc:407] Connection header received
0926 14:55:21.951368 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.951370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.951386 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.951462 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:21.951688 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.952718 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.953813 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.954433 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.954495 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.954546 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.954749 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.954823 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260926 14:55:21.955338 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.950967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35187 (local address 127.0.0.1:44672)
0926 14:55:21.951225 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:21.951235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:21.951297 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:21.951554 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:21.951554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:21.951556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:21.951606 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.951607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.952865 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.952866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.954082 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.954085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.955083 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.955084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.955149 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.955152 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:21.955230 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:55:21.955268 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260926 14:55:21.955358 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.951084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.5.15.130:35187)
0926 14:55:21.951253 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:21.951256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:21.951283 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:55:21.951386 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:21.951388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:21.951401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:21.951483 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:21.951683 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.952719 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.954160 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.954928 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.954997 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.955048 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:21.955250 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 14:55:21.955289 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260926 14:55:21.955516 23830 tablet_service.cc:1467] Processing CreateTablet for tablet a46153e313804427b503fb1406bb638a (DEFAULT_TABLE table=test-table [id=c1e159bb4ddd4160a522b6c4af0b9e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:21.955768 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:21.950715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:44089 (local address 127.0.0.1:52064)
0926 14:55:21.951118 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:21.951156 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:21.951262 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:21.951540 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:21.951543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:21.951545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:21.951605 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.951607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.952835 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.952836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.953736 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:21.953738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:21.954576 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:21.954577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:21.954636 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:21.954639 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:21.955384 (+ 745us) client_negotiation.cc:240] Negotiation successful
0926 14:55:21.955646 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":35,"thread_start_us":75,"threads_started":1}
I20260926 14:55:21.955870 23830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46153e313804427b503fb1406bb638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:21.956065 23964 tablet_service.cc:1467] Processing CreateTablet for tablet a46153e313804427b503fb1406bb638a (DEFAULT_TABLE table=test-table [id=c1e159bb4ddd4160a522b6c4af0b9e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:21.956393 23964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46153e313804427b503fb1406bb638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:21.956521 24098 tablet_service.cc:1467] Processing CreateTablet for tablet a46153e313804427b503fb1406bb638a (DEFAULT_TABLE table=test-table [id=c1e159bb4ddd4160a522b6c4af0b9e60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:21.956794 24098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46153e313804427b503fb1406bb638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:21.958495 24178 tablet_bootstrap.cc:492] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: Bootstrap starting.
I20260926 14:55:21.959156 24178 tablet_bootstrap.cc:654] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:21.959674 24179 tablet_bootstrap.cc:492] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Bootstrap starting.
I20260926 14:55:21.959734 24178 log.cc:824] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:21.960350 24179 tablet_bootstrap.cc:654] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:21.960651 24178 tablet_bootstrap.cc:492] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: No bootstrap required, opened a new log
I20260926 14:55:21.960741 24178 ts_tablet_manager.cc:1397] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:21.960896 24179 log.cc:824] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:21.961587 24179 tablet_bootstrap.cc:492] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: No bootstrap required, opened a new log
I20260926 14:55:21.961658 24179 ts_tablet_manager.cc:1397] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:21.962776 24178 raft_consensus.cc:348] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:21.962906 24181 tablet_bootstrap.cc:492] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Bootstrap starting.
I20260926 14:55:21.962949 24178 raft_consensus.cc:374] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:21.962988 24178 raft_consensus.cc:729] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd8d4f1cdfb4202863111465c624576, State: Initialized, Role: FOLLOWER
I20260926 14:55:21.963101 24178 consensus_queue.cc:260] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:21.963352 24178 ts_tablet_manager.cc:1428] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 14:55:21.963469 23896 heartbeater.cc:503] Master 127.5.15.190:35613 was elected leader, sending a full tablet report...
I20260926 14:55:21.963541 24181 tablet_bootstrap.cc:654] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:21.963675 24179 raft_consensus.cc:348] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:21.963843 24179 raft_consensus.cc:374] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:21.963881 24179 raft_consensus.cc:729] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46aff6972a0a4214af531a56c46149e8, State: Initialized, Role: FOLLOWER
I20260926 14:55:21.963997 24179 consensus_queue.cc:260] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:21.964144 24181 log.cc:824] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:21.964267 24179 ts_tablet_manager.cc:1428] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:21.964411 24030 heartbeater.cc:503] Master 127.5.15.190:35613 was elected leader, sending a full tablet report...
I20260926 14:55:21.965114 24181 tablet_bootstrap.cc:492] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: No bootstrap required, opened a new log
I20260926 14:55:21.965184 24181 ts_tablet_manager.cc:1397] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:21.966416 24181 raft_consensus.cc:348] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:21.966562 24181 raft_consensus.cc:374] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:21.966604 24181 raft_consensus.cc:729] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdb951ba7d940b38ea332dacecd85a1, State: Initialized, Role: FOLLOWER
I20260926 14:55:21.966703 24181 consensus_queue.cc:260] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:21.967005 24181 ts_tablet_manager.cc:1428] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:55:21.967034 24164 heartbeater.cc:503] Master 127.5.15.190:35613 was elected leader, sending a full tablet report...
W20260926 14:55:22.037935 24031 tablet.cc:2369] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:55:22.126921 23897 tablet.cc:2369] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:55:22.179148 24165 tablet.cc:2369] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:55:22.196637 24183 raft_consensus.cc:482] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:22.196779 24183 raft_consensus.cc:504] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:22.197206 24183 leader_election.cc:288] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecdb951ba7d940b38ea332dacecd85a1 (127.5.15.131:44089), 46aff6972a0a4214af531a56c46149e8 (127.5.15.130:35187)
I20260926 14:55:22.200198 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.197485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:55347 (local address 127.5.15.131:44089)
0926 14:55:22.197543 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.197545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.197592 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:55:22.197677 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.197678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.197687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.197771 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.197934 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.198427 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.199482 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.199860 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.199942 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.199950 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.200095 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.200128 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:55:22.200198 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.197463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47779 (local address 127.5.15.130:35187)
0926 14:55:22.197538 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.197541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.197628 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:55:22.197695 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.197696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.197709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.197782 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.197961 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.198433 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.199473 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.199850 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.199937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.199946 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.200090 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.200126 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:55:22.200220 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.197404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35187 (local address 127.5.15.129:47779)
0926 14:55:22.197562 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.197573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.197637 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.197785 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.197786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.197788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.197889 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.197890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.198519 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.198520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.199378 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.199380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.199936 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.199938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.200008 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.200012 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.200086 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.200133 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":30,"thread_start_us":62,"threads_started":1}
I20260926 14:55:22.200232 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.197385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:44089 (local address 127.5.15.129:55347)
0926 14:55:22.197530 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.197541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.197608 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.197775 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.197776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.197778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.197859 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.197865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.198509 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.198510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.199389 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.199391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.199942 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.199943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.200018 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.200021 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.200088 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.200130 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":40,"thread_start_us":87,"threads_started":1}
I20260926 14:55:22.200587 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46153e313804427b503fb1406bb638a" candidate_uuid: "6bd8d4f1cdfb4202863111465c624576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb951ba7d940b38ea332dacecd85a1" is_pre_election: true
I20260926 14:55:22.200641 23984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46153e313804427b503fb1406bb638a" candidate_uuid: "6bd8d4f1cdfb4202863111465c624576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aff6972a0a4214af531a56c46149e8" is_pre_election: true
I20260926 14:55:22.200786 24118 raft_consensus.cc:2454] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd8d4f1cdfb4202863111465c624576 in term 0.
I20260926 14:55:22.200855 23984 raft_consensus.cc:2454] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd8d4f1cdfb4202863111465c624576 in term 0.
I20260926 14:55:22.201042 23783 leader_election.cc:302] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd8d4f1cdfb4202863111465c624576, ecdb951ba7d940b38ea332dacecd85a1; no voters:
I20260926 14:55:22.201207 24183 raft_consensus.cc:2781] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:55:22.201272 24183 raft_consensus.cc:482] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:55:22.201300 24183 raft_consensus.cc:3037] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.201993 24183 raft_consensus.cc:504] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:22.202129 24183 leader_election.cc:288] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [CANDIDATE]: Term 1 election: Requested vote from peers ecdb951ba7d940b38ea332dacecd85a1 (127.5.15.131:44089), 46aff6972a0a4214af531a56c46149e8 (127.5.15.130:35187)
I20260926 14:55:22.202283 24118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46153e313804427b503fb1406bb638a" candidate_uuid: "6bd8d4f1cdfb4202863111465c624576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecdb951ba7d940b38ea332dacecd85a1"
I20260926 14:55:22.202338 23984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46153e313804427b503fb1406bb638a" candidate_uuid: "6bd8d4f1cdfb4202863111465c624576" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46aff6972a0a4214af531a56c46149e8"
I20260926 14:55:22.202363 24118 raft_consensus.cc:3037] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.202436 23984 raft_consensus.cc:3037] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.203118 24118 raft_consensus.cc:2454] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd8d4f1cdfb4202863111465c624576 in term 1.
I20260926 14:55:22.203243 23984 raft_consensus.cc:2454] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd8d4f1cdfb4202863111465c624576 in term 1.
I20260926 14:55:22.203291 23783 leader_election.cc:302] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd8d4f1cdfb4202863111465c624576, ecdb951ba7d940b38ea332dacecd85a1; no voters:
I20260926 14:55:22.203379 24183 raft_consensus.cc:2781] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:22.203553 24183 raft_consensus.cc:686] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd8d4f1cdfb4202863111465c624576, State: Running, Role: LEADER
I20260926 14:55:22.203670 24183 consensus_queue.cc:237] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } }
I20260926 14:55:22.204525 23704 catalog_manager.cc:5495] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bd8d4f1cdfb4202863111465c624576 (127.5.15.129). New cstate: current_term: 1 leader_uuid: "6bd8d4f1cdfb4202863111465c624576" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd8d4f1cdfb4202863111465c624576" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 42089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 } health_report { overall_health: UNKNOWN } } }
I20260926 14:55:22.237705 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.235010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.5.15.129:42089)
0926 14:55:22.235054 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.235057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.235068 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:55:22.235109 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.235110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.235132 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.235226 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:22.235399 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.235868 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.236685 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.236959 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.237032 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.237244 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:22.237371 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:22.237451 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.237559 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.237604 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:55:22.239886 24118 raft_consensus.cc:1261] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 1 FOLLOWER]: Refusing update from remote peer 6bd8d4f1cdfb4202863111465c624576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:22.239987 23984 raft_consensus.cc:1261] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 1 FOLLOWER]: Refusing update from remote peer 6bd8d4f1cdfb4202863111465c624576: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:22.240223 24183 consensus_queue.cc:1035] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecdb951ba7d940b38ea332dacecd85a1" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 44089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:22.240352 24189 consensus_queue.cc:1035] T a46153e313804427b503fb1406bb638a P 6bd8d4f1cdfb4202863111465c624576 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46aff6972a0a4214af531a56c46149e8" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:22.242869 24194 mvcc.cc:205] Tried to move back new op lower bound from 7333619803091898368 to 7333619802946977792. Current Snapshot: MvccSnapshot[applied={T|T < 7333619803091898368}]
I20260926 14:55:22.242969 24193 mvcc.cc:205] Tried to move back new op lower bound from 7333619803091898368 to 7333619802946977792. Current Snapshot: MvccSnapshot[applied={T|T < 7333619803091898368}]
I20260926 14:55:22.244834 23705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:22.244939 23705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:22.245659 23705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 362e07f4f10241a188e436b3090aa6a9: Sending DeleteTablet for 3 replicas of tablet a46153e313804427b503fb1406bb638a
I20260926 14:55:22.246011 24098 tablet_service.cc:1514] Processing DeleteTablet for tablet a46153e313804427b503fb1406bb638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:22 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:52064
I20260926 14:55:22.246241 24202 tablet_replica.cc:330] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: stopping tablet replica
I20260926 14:55:22.246230 23964 tablet_service.cc:1514] Processing DeleteTablet for tablet a46153e313804427b503fb1406bb638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:22 UTC) from {username='kudu', principal='kudu/127.5.15.190@KRBTEST.COM'} at 127.0.0.1:44672
I20260926 14:55:22.246338 24202 raft_consensus.cc:2229] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:22.246466 24202 raft_consensus.cc:2258] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:22.246659 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23767
I20260926 14:55:22.246896 24202 ts_tablet_manager.cc:1905] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:22.248343 24203 tablet_replica.cc:330] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: stopping tablet replica
I20260926 14:55:22.248469 24203 raft_consensus.cc:2229] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:22.248607 24203 raft_consensus.cc:2258] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:22.248919 24202 ts_tablet_manager.cc:1918] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:22.248993 24202 log.cc:1196] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/a46153e313804427b503fb1406bb638a
I20260926 14:55:22.249305 24202 ts_tablet_manager.cc:1939] T a46153e313804427b503fb1406bb638a P ecdb951ba7d940b38ea332dacecd85a1: Deleting consensus metadata
I20260926 14:55:22.249716 23689 catalog_manager.cc:4841] TS ecdb951ba7d940b38ea332dacecd85a1 (127.5.15.131:44089): tablet a46153e313804427b503fb1406bb638a (table test-table [id=c1e159bb4ddd4160a522b6c4af0b9e60]) successfully deleted
W20260926 14:55:22.252537 23692 connection.cc:535] client connection to 127.5.15.129:42089 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 14:55:22.252627 23692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:55:22.252808 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23901
I20260926 14:55:22.253001 24203 ts_tablet_manager.cc:1905] T a46153e313804427b503fb1406bb638a P 46aff6972a0a4214af531a56c46149e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:22.253455 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.253349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:42089 (local address 127.0.0.1:33146)
0926 14:55:22.253386 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.253420 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:42089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 14:55:22.253571 23692 catalog_manager.cc:4584] TS 6bd8d4f1cdfb4202863111465c624576 (127.5.15.129:42089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a46153e313804427b503fb1406bb638a: Network error: Client connection negotiation failed: client connection to 127.5.15.129:42089: connect: Connection refused (error 111)
I20260926 14:55:22.258262 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24035
I20260926 14:55:22.258291 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.258162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:35187 (local address 127.0.0.1:44688)
0926 14:55:22.258230 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.258261 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.130:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:55:22.262792 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 23673
2026-09-26T14:55:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2087 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T14:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:22Z Disabled control of system clock
I20260926 14:55:22.289553 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36599
--webserver_interface=127.5.15.190
--webserver_port=0
--builtin_ntp_servers=127.5.15.148:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:22.370158 24208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:22.370369 24208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:22.370388 24208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:22.371479 24208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:22.371519 24208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:22.371531 24208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:22.371543 24208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:22.371554 24208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:22.372769 24208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:22.372998 24208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:22.373221 24208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:22.377677 24216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.377688 24213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.377753 24214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:22.377926 24208 server_base.cc:1034] running on GCE node
I20260926 14:55:22.378259 24208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:22.378557 24208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:22.379716 24208 hybrid_clock.cc:648] HybridClock initialized: now 1790434522379674 us; error 54 us; skew 500 ppm
I20260926 14:55:22.380846 24208 webserver.cc:460] Webserver started at http://127.5.15.190:44543/ using document root <none> and password file <none>
I20260926 14:55:22.381102 24208 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:22.381171 24208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:22.381327 24208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:22.382404 24208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "14a91a4eed204e6295c6815ead9c87f6"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "8fb0c6fb9f4cb2a8a55c2d2a961f8bc9"
server_key_iv: "211beca5f35e64540fa2eb9ac4cfefd7"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.382853 24208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "14a91a4eed204e6295c6815ead9c87f6"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "8fb0c6fb9f4cb2a8a55c2d2a961f8bc9"
server_key_iv: "211beca5f35e64540fa2eb9ac4cfefd7"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.384145 24208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:22.384837 24222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.384994 24208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:22.385087 24208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "14a91a4eed204e6295c6815ead9c87f6"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "8fb0c6fb9f4cb2a8a55c2d2a961f8bc9"
server_key_iv: "211beca5f35e64540fa2eb9ac4cfefd7"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.385169 24208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:22.408952 24208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:22.409281 24208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:22.409439 24208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:22.413733 24208 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:36599
I20260926 14:55:22.413786 24274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:36599 every 8 connection(s)
I20260926 14:55:22.414198 24208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:22.415191 24275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:22.417328 24275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6: Bootstrap starting.
I20260926 14:55:22.417872 24275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:22.418167 24275 log.cc:824] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:22.418874 24275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6: No bootstrap required, opened a new log
I20260926 14:55:22.420398 24275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a91a4eed204e6295c6815ead9c87f6" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36599 } }
I20260926 14:55:22.420526 24275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:22.420568 24275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a91a4eed204e6295c6815ead9c87f6, State: Initialized, Role: FOLLOWER
I20260926 14:55:22.420715 24275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a91a4eed204e6295c6815ead9c87f6" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36599 } }
I20260926 14:55:22.420790 24275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:22.420831 24275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:22.420878 24275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.421330 24275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a91a4eed204e6295c6815ead9c87f6" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36599 } }
I20260926 14:55:22.421461 24275 leader_election.cc:302] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14a91a4eed204e6295c6815ead9c87f6; no voters:
I20260926 14:55:22.421635 24275 leader_election.cc:288] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:22.421700 24278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:22.421864 24278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [term 1 LEADER]: Becoming Leader. State: Replica: 14a91a4eed204e6295c6815ead9c87f6, State: Running, Role: LEADER
I20260926 14:55:22.421898 24275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:22.421958 24278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a91a4eed204e6295c6815ead9c87f6" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36599 } }
I20260926 14:55:22.422394 24280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14a91a4eed204e6295c6815ead9c87f6. Latest consensus state: current_term: 1 leader_uuid: "14a91a4eed204e6295c6815ead9c87f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a91a4eed204e6295c6815ead9c87f6" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36599 } } }
I20260926 14:55:22.422490 24280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:22.422497 24279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14a91a4eed204e6295c6815ead9c87f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14a91a4eed204e6295c6815ead9c87f6" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36599 } } }
I20260926 14:55:22.422609 24279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:22.422705 24287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:22.423197 24287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:22.423614 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24208
I20260926 14:55:22.423735 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:22.423918 5182 external_mini_cluster.cc:1428] Setting key a59aecd1b56698828f760700bc35a1e3
I20260926 14:55:22.424778 24287 catalog_manager.cc:1347] Generated new cluster ID: 524c024458d44218bb393acd466ade24
I20260926 14:55:22.424831 24287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:22.429718 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.424538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.5.15.190:36599)
0926 14:55:22.424709 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.424717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.424746 (+ 29us) server_negotiation.cc:407] Connection header received
0926 14:55:22.424856 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.424873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.424951 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.425090 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:22.425316 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.426615 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.427637 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.428265 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.428384 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.429014 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:22.429017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:22.429018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:22.429237 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:22.429277 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:22.429292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:22.429362 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.429429 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.429510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260926 14:55:22.432832 24287 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:22.433499 24287 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:22.440712 24287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6: Generated new TSK 0
I20260926 14:55:22.440884 24287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:22.447263 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.129:0
--local_ip_for_outbound_sockets=127.5.15.129
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36599
--builtin_ntp_servers=127.5.15.148:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:55:22.533828 24299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:22.534020 24299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:22.534041 24299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:22.535147 24299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:22.535187 24299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:22.535211 24299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.129
I20260926 14:55:22.536460 24299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.15.129
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.129
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:22.536675 24299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:22.536962 24299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:22.541702 24305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.541745 24304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.541735 24307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:22.542045 24299 server_base.cc:1034] running on GCE node
I20260926 14:55:22.542243 24299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:22.542562 24299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:22.543747 24299 hybrid_clock.cc:648] HybridClock initialized: now 1790434522543687 us; error 77 us; skew 500 ppm
I20260926 14:55:22.545083 24299 webserver.cc:460] Webserver started at http://127.5.15.129:36631/ using document root <none> and password file <none>
I20260926 14:55:22.545367 24299 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:22.545434 24299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:22.545563 24299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:22.546557 24299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data/instance:
uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "b967607d89a318384321544818901d83"
server_key_iv: "9e3d4c23d1165afd79318498cdd55c4f"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.546932 24299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "b967607d89a318384321544818901d83"
server_key_iv: "9e3d4c23d1165afd79318498cdd55c4f"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.548198 24299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:22.548947 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.549192 24299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:22.549284 24299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal
uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "b967607d89a318384321544818901d83"
server_key_iv: "9e3d4c23d1165afd79318498cdd55c4f"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.549350 24299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:22.570848 24299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:22.571219 24299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:22.571347 24299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:22.571538 24299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:22.571818 24299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:22.571849 24299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.571871 24299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:22.571883 24299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.577550 24299 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.129:36291
I20260926 14:55:22.577642 24426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.129:36291 every 8 connection(s)
I20260926 14:55:22.577975 24299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/data/info.pb
I20260926 14:55:22.582535 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24299
I20260926 14:55:22.582634 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-0/wal/instance
I20260926 14:55:22.582847 5182 external_mini_cluster.cc:1428] Setting key 934d4a57a3893212690b7e6232ba37a9
I20260926 14:55:22.583168 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.578726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.129:47551 (local address 127.5.15.190:36599)
0926 14:55:22.578789 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.578791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.579048 (+ 257us) server_negotiation.cc:407] Connection header received
0926 14:55:22.579511 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.579512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.579524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.579573 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:22.580208 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.580701 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.581656 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.581863 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.581923 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.582743 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:22.582766 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:22.582766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:22.582774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:22.582790 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:22.582795 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:22.582853 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.583042 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.583078 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:55:22.583374 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.578697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36599 (local address 127.5.15.129:47551)
0926 14:55:22.578921 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.578955 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.579430 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.579636 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.579639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.579647 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:22.580141 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.580145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.580785 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.580786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.581562 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.581564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.581947 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.581948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.582057 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.582570 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:22.582571 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:22.582614 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:22.582646 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:22.582879 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:22.582885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:22.582889 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.583003 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.583105 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":126,"threads_started":1}
I20260926 14:55:22.583799 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.130:0
--local_ip_for_outbound_sockets=127.5.15.130
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36599
--builtin_ntp_servers=127.5.15.148:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:55:22.584066 24427 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36599
I20260926 14:55:22.584160 24427 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:22.584354 24427 heartbeater.cc:511] Master 127.5.15.190:36599 requested a full tablet report, sending...
I20260926 14:55:22.584821 24239 ts_manager.cc:194] Registered new tserver with Master: b4c32ce06b5f4f6f9db6e4671d8fde53 (127.5.15.129:36291)
I20260926 14:55:22.585435 24239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.129:47551
W20260926 14:55:22.663327 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:22.663520 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:22.663542 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:22.664633 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:22.664664 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:22.664687 24430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.130
I20260926 14:55:22.665944 24430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.15.130
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.130
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:22.666112 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:22.666342 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:22.671036 24435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.671007 24436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.671034 24438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:22.671222 24430 server_base.cc:1034] running on GCE node
I20260926 14:55:22.671454 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:22.671690 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:22.672840 24430 hybrid_clock.cc:648] HybridClock initialized: now 1790434522672830 us; error 26 us; skew 500 ppm
I20260926 14:55:22.674032 24430 webserver.cc:460] Webserver started at http://127.5.15.130:35291/ using document root <none> and password file <none>
I20260926 14:55:22.674269 24430 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:22.674314 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:22.674463 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:22.675539 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data/instance:
uuid: "178d35819e794826b82c561b9d269b16"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "fdc746920a17cbc81199c629407592d1"
server_key_iv: "fdefbd4b0f5f4212679ed3f85a777ddd"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.675887 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance:
uuid: "178d35819e794826b82c561b9d269b16"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "fdc746920a17cbc81199c629407592d1"
server_key_iv: "fdefbd4b0f5f4212679ed3f85a777ddd"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.677138 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 14:55:22.677794 24444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.677974 24430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:22.678053 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal
uuid: "178d35819e794826b82c561b9d269b16"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "fdc746920a17cbc81199c629407592d1"
server_key_iv: "fdefbd4b0f5f4212679ed3f85a777ddd"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.678103 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:22.699502 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:22.699770 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:22.699874 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:22.700055 24430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:22.700392 24430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:22.700428 24430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.700484 24430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:22.700510 24430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.705947 24430 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.130:36907
I20260926 14:55:22.706038 24557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.130:36907 every 8 connection(s)
I20260926 14:55:22.706341 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/data/info.pb
I20260926 14:55:22.708714 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24430
I20260926 14:55:22.708834 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal/instance
I20260926 14:55:22.709010 5182 external_mini_cluster.cc:1428] Setting key d7ed6cb8203de1e23bb3ec036a5fb8fb
I20260926 14:55:22.710005 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.15.131:0
--local_ip_for_outbound_sockets=127.5.15.131
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36599
--builtin_ntp_servers=127.5.15.148:38305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:55:22.712319 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.707109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.130:45399 (local address 127.5.15.190:36599)
0926 14:55:22.707176 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.707177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.707410 (+ 233us) server_negotiation.cc:407] Connection header received
0926 14:55:22.707951 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.707952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.707965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.708018 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:22.708675 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.709389 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.710829 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.711021 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.711078 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.712060 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:22.712061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:22.712062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:22.712067 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:22.712083 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:22.712087 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:22.712130 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.712236 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.712261 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:55:22.712571 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.707082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36599 (local address 127.5.15.130:45399)
0926 14:55:22.707289 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.707322 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.707832 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.708085 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.708089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.708098 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:22.708580 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.708585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.709481 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.709482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.710711 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.710714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.711112 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.711113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.711277 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.711905 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:22.711906 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:22.711934 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:22.711956 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:22.712141 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:22.712146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:22.712148 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.712217 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.712324 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260926 14:55:22.713083 24558 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36599
I20260926 14:55:22.713168 24558 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:22.713349 24558 heartbeater.cc:511] Master 127.5.15.190:36599 requested a full tablet report, sending...
I20260926 14:55:22.713692 24239 ts_manager.cc:194] Registered new tserver with Master: 178d35819e794826b82c561b9d269b16 (127.5.15.130:36907)
I20260926 14:55:22.714052 24239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.130:45399
W20260926 14:55:22.789737 24561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:22.789940 24561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:22.789991 24561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:22.791258 24561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:22.791317 24561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:22.791373 24561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.15.131
I20260926 14:55:22.792670 24561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.15.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.15.131
--webserver_port=0
--tserver_master_addrs=127.5.15.190:36599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.15.131
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:22.792900 24561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:22.793165 24561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:22.797713 24569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:22.797744 24566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:22.797766 24561 server_base.cc:1034] running on GCE node
W20260926 14:55:22.797750 24567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:22.798108 24561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:22.798381 24561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:22.799515 24561 hybrid_clock.cc:648] HybridClock initialized: now 1790434522799505 us; error 28 us; skew 500 ppm
I20260926 14:55:22.800606 24561 webserver.cc:460] Webserver started at http://127.5.15.131:33607/ using document root <none> and password file <none>
I20260926 14:55:22.800856 24561 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:22.800904 24561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:22.801076 24561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:22.802040 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data/instance:
uuid: "29687b5c1764483dbe09bf8886562735"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "216dc51f01d9ae348506a5db97e7bf5a"
server_key_iv: "e3909bd974430edc1403be6ec710d16f"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.802419 24561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance:
uuid: "29687b5c1764483dbe09bf8886562735"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "216dc51f01d9ae348506a5db97e7bf5a"
server_key_iv: "e3909bd974430edc1403be6ec710d16f"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.803673 24561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:22.804338 24575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.804488 24561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:22.804565 24561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal
uuid: "29687b5c1764483dbe09bf8886562735"
format_stamp: "Formatted at 2026-09-26 14:55:22 on dist-test-slave-s4p1"
server_key: "216dc51f01d9ae348506a5db97e7bf5a"
server_key_iv: "e3909bd974430edc1403be6ec710d16f"
server_key_version: "encryptionkey@0"
I20260926 14:55:22.804617 24561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:22.829633 24561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:22.829890 24561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:22.829989 24561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:22.830168 24561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:55:22.830506 24561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:55:22.830538 24561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.830590 24561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:55:22.830616 24561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:22.836148 24561 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.131:43747
I20260926 14:55:22.836195 24688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.131:43747 every 8 connection(s)
I20260926 14:55:22.836525 24561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/data/info.pb
I20260926 14:55:22.841363 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.837196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:52841 (local address 127.5.15.190:36599)
0926 14:55:22.837264 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.837266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.837514 (+ 248us) server_negotiation.cc:407] Connection header received
0926 14:55:22.837972 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.837973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.837982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.838028 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:22.838671 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.839224 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.840129 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.840305 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.840352 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.841142 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:22.841142 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:22.841143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:22.841146 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:22.841156 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:22.841159 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:22.841196 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.841286 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.841308 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:55:22.841578 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.837167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36599 (local address 127.5.15.131:52841)
0926 14:55:22.837386 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.837418 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.837889 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.838105 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.838109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.838127 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:22.838600 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.838604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.839305 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.839306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.840055 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.840057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.840382 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.840383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.840485 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.841024 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:55:22.841024 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:22.841051 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:55:22.841072 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:22.841217 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:22.841221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:22.841223 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.841275 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.841349 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260926 14:55:22.842111 24689 heartbeater.cc:347] Connected to a master server at 127.5.15.190:36599
I20260926 14:55:22.842219 24689 heartbeater.cc:464] Registering TS with master...
I20260926 14:55:22.842399 24689 heartbeater.cc:511] Master 127.5.15.190:36599 requested a full tablet report, sending...
I20260926 14:55:22.842792 24239 ts_manager.cc:194] Registered new tserver with Master: 29687b5c1764483dbe09bf8886562735 (127.5.15.131:43747)
I20260926 14:55:22.843139 24239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.15.131:52841
I20260926 14:55:22.845268 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24561
I20260926 14:55:22.845338 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal/instance
I20260926 14:55:22.845539 5182 external_mini_cluster.cc:1428] Setting key 0b47ef352bf3841eaf2c8ff1bdcd9570
I20260926 14:55:22.846390 5182 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:55:22.850689 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.847690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.5.15.190:36599)
0926 14:55:22.847778 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.847780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.847877 (+ 97us) server_negotiation.cc:407] Connection header received
0926 14:55:22.847949 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.847950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.847969 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.848030 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:22.848190 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.848641 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.849408 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.849589 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.849676 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.850352 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:22.850353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:22.850353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:22.850357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:22.850370 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:22.850373 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:22.850488 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.850562 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.850584 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:55:22.852108 24239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:55:22.853718 24239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:55:22.859824 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.855740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36291 (local address 127.0.0.1:40350)
0926 14:55:22.856056 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.856067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.856124 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.856330 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.856333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.856335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.856399 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.856400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.857456 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.857457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.858627 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.858630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.859543 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.859545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.859631 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.859645 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.859714 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.859759 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":73,"threads_started":1}
I20260926 14:55:22.859982 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.855699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:36907 (local address 127.0.0.1:41028)
0926 14:55:22.855827 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.855865 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.855927 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.856427 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.856428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.856430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.856461 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.856462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.857758 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.857758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.858857 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.858866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.859726 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.859727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.859793 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.859797 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.859854 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.859893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":7,"thread_start_us":81,"threads_started":1}
I20260926 14:55:22.860088 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.855884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.5.15.130:36907)
0926 14:55:22.856249 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.856254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.856264 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:22.856302 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.856304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.856317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.856426 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.856529 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.857672 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.858941 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.859655 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.859719 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.859796 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.859868 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.859999 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":85,"threads_started":1}
I20260926 14:55:22.860414 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.855841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.5.15.129:36291)
0926 14:55:22.856035 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.856040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.856121 (+ 81us) server_negotiation.cc:407] Connection header received
0926 14:55:22.856209 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.856212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.856227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.856321 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.856632 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.857373 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.858708 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.859451 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.859532 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.859618 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.860282 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.860329 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260926 14:55:22.860781 24492 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa49be235364a568e2bdab5a025f617 (DEFAULT_TABLE table=test-table [id=f5939cd33b63410398411af7c2d5af3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:22.861130 24492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa49be235364a568e2bdab5a025f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:22.861136 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.855929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.131:43747 (local address 127.0.0.1:38550)
0926 14:55:22.856088 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.856098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.856148 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.856450 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.856451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.856453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.856500 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.856502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.858785 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.858785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.859541 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.859542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.860910 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.860911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.860989 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.860993 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.861042 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.861076 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260926 14:55:22.861064 24361 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa49be235364a568e2bdab5a025f617 (DEFAULT_TABLE table=test-table [id=f5939cd33b63410398411af7c2d5af3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:22.861374 24361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa49be235364a568e2bdab5a025f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:22.861438 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.856093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.5.15.131:43747)
0926 14:55:22.856255 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.856259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.856267 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:55:22.856301 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.856303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.856317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.856395 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.856576 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.857669 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.859632 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.860738 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.860800 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.860884 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.861315 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.861352 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260926 14:55:22.861948 24623 tablet_service.cc:1467] Processing CreateTablet for tablet 6fa49be235364a568e2bdab5a025f617 (DEFAULT_TABLE table=test-table [id=f5939cd33b63410398411af7c2d5af3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:55:22.862231 24623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fa49be235364a568e2bdab5a025f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:22.864594 24703 tablet_bootstrap.cc:492] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53: Bootstrap starting.
I20260926 14:55:22.864737 24705 tablet_bootstrap.cc:492] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Bootstrap starting.
I20260926 14:55:22.864769 24704 tablet_bootstrap.cc:492] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Bootstrap starting.
I20260926 14:55:22.865268 24705 tablet_bootstrap.cc:654] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:22.865283 24704 tablet_bootstrap.cc:654] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:22.865559 24705 log.cc:824] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:22.865598 24704 log.cc:824] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:22.866309 24705 tablet_bootstrap.cc:492] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: No bootstrap required, opened a new log
I20260926 14:55:22.866309 24704 tablet_bootstrap.cc:492] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: No bootstrap required, opened a new log
I20260926 14:55:22.866379 24705 ts_tablet_manager.cc:1397] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:22.866379 24704 ts_tablet_manager.cc:1397] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:55:22.866391 24703 tablet_bootstrap.cc:654] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:22.866667 24703 log.cc:824] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:22.867592 24703 tablet_bootstrap.cc:492] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53: No bootstrap required, opened a new log
I20260926 14:55:22.867667 24703 ts_tablet_manager.cc:1397] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:55:22.868956 24705 raft_consensus.cc:348] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.869123 24705 raft_consensus.cc:374] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:22.869073 24704 raft_consensus.cc:348] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.869163 24705 raft_consensus.cc:729] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29687b5c1764483dbe09bf8886562735, State: Initialized, Role: FOLLOWER
I20260926 14:55:22.869228 24704 raft_consensus.cc:374] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:22.869259 24704 raft_consensus.cc:729] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178d35819e794826b82c561b9d269b16, State: Initialized, Role: FOLLOWER
I20260926 14:55:22.869326 24705 consensus_queue.cc:260] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.869375 24704 consensus_queue.cc:260] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.869580 24705 ts_tablet_manager.cc:1428] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:22.869588 24704 ts_tablet_manager.cc:1428] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 14:55:22.869503 24703 raft_consensus.cc:348] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.869635 24558 heartbeater.cc:503] Master 127.5.15.190:36599 was elected leader, sending a full tablet report...
I20260926 14:55:22.869654 24703 raft_consensus.cc:374] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:22.869686 24703 raft_consensus.cc:729] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c32ce06b5f4f6f9db6e4671d8fde53, State: Initialized, Role: FOLLOWER
I20260926 14:55:22.869702 24689 heartbeater.cc:503] Master 127.5.15.190:36599 was elected leader, sending a full tablet report...
I20260926 14:55:22.869792 24703 consensus_queue.cc:260] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.870019 24703 ts_tablet_manager.cc:1428] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 14:55:22.870160 24427 heartbeater.cc:503] Master 127.5.15.190:36599 was elected leader, sending a full tablet report...
W20260926 14:55:22.956907 24559 tablet.cc:2369] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:55:22.958727 24709 raft_consensus.cc:482] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:55:22.958839 24709 raft_consensus.cc:504] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.959277 24709 leader_election.cc:288] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 178d35819e794826b82c561b9d269b16 (127.5.15.130:36907), b4c32ce06b5f4f6f9db6e4671d8fde53 (127.5.15.129:36291)
I20260926 14:55:22.963183 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.959531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:43233 (local address 127.5.15.129:36291)
0926 14:55:22.959593 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.959595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.959604 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:22.959651 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.959652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.959665 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.959759 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.959978 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.960719 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.962261 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.962842 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.962918 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.962925 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.963078 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.963116 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:55:22.963195 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.959555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.15.131:42227 (local address 127.5.15.130:36907)
0926 14:55:22.959633 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.959635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.959713 (+ 78us) server_negotiation.cc:407] Connection header received
0926 14:55:22.959741 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.959742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.959754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.959811 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:55:22.959973 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.960710 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.962261 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.962841 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.962917 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.962925 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.963078 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.963143 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:55:22.963214 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.959457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36291 (local address 127.5.15.131:43233)
0926 14:55:22.959518 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.959531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.959574 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.959783 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.959784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.959802 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.959892 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.959894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.960794 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.960795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.962149 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.962152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.962923 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.962924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.962997 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.963003 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.963060 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.963111 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":99}
I20260926 14:55:22.963227 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.959457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.130:36907 (local address 127.5.15.131:42227)
0926 14:55:22.959632 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.959642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:22.959697 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:22.959825 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:22.959825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:22.959827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:55:22.959892 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.959894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.960794 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.960795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.962149 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:22.962152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.962936 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.962937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:22.962997 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.963004 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 14:55:22.963060 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:55:22.963110 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":86,"thread_start_us":93,"threads_started":1}
I20260926 14:55:22.963586 24381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa49be235364a568e2bdab5a025f617" candidate_uuid: "29687b5c1764483dbe09bf8886562735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" is_pre_election: true
I20260926 14:55:22.963764 24381 raft_consensus.cc:2454] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29687b5c1764483dbe09bf8886562735 in term 0.
I20260926 14:55:22.963757 24512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa49be235364a568e2bdab5a025f617" candidate_uuid: "29687b5c1764483dbe09bf8886562735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178d35819e794826b82c561b9d269b16" is_pre_election: true
I20260926 14:55:22.963912 24512 raft_consensus.cc:2454] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29687b5c1764483dbe09bf8886562735 in term 0.
I20260926 14:55:22.963999 24576 leader_election.cc:302] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29687b5c1764483dbe09bf8886562735, b4c32ce06b5f4f6f9db6e4671d8fde53; no voters:
I20260926 14:55:22.964151 24709 raft_consensus.cc:2781] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:55:22.964205 24709 raft_consensus.cc:482] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:55:22.964226 24709 raft_consensus.cc:3037] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.964916 24709 raft_consensus.cc:504] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.965054 24709 leader_election.cc:288] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [CANDIDATE]: Term 1 election: Requested vote from peers 178d35819e794826b82c561b9d269b16 (127.5.15.130:36907), b4c32ce06b5f4f6f9db6e4671d8fde53 (127.5.15.129:36291)
I20260926 14:55:22.965178 24512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa49be235364a568e2bdab5a025f617" candidate_uuid: "29687b5c1764483dbe09bf8886562735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178d35819e794826b82c561b9d269b16"
I20260926 14:55:22.965194 24381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fa49be235364a568e2bdab5a025f617" candidate_uuid: "29687b5c1764483dbe09bf8886562735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53"
I20260926 14:55:22.965256 24512 raft_consensus.cc:3037] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.965256 24381 raft_consensus.cc:3037] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:22.966125 24512 raft_consensus.cc:2454] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29687b5c1764483dbe09bf8886562735 in term 1.
I20260926 14:55:22.966135 24381 raft_consensus.cc:2454] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29687b5c1764483dbe09bf8886562735 in term 1.
I20260926 14:55:22.966308 24577 leader_election.cc:302] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178d35819e794826b82c561b9d269b16, 29687b5c1764483dbe09bf8886562735; no voters:
I20260926 14:55:22.966408 24709 raft_consensus.cc:2781] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:22.966534 24709 raft_consensus.cc:686] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 1 LEADER]: Becoming Leader. State: Replica: 29687b5c1764483dbe09bf8886562735, State: Running, Role: LEADER
I20260926 14:55:22.966620 24709 consensus_queue.cc:237] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } }
I20260926 14:55:22.967446 24239 catalog_manager.cc:5495] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29687b5c1764483dbe09bf8886562735 (127.5.15.131). New cstate: current_term: 1 leader_uuid: "29687b5c1764483dbe09bf8886562735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29687b5c1764483dbe09bf8886562735" member_type: VOTER last_known_addr { host: "127.5.15.131" port: 43747 } health_report { overall_health: HEALTHY } } }
I20260926 14:55:22.977769 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.975501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.5.15.131:43747)
0926 14:55:22.975541 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:22.975543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:22.975552 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:22.975604 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:22.975605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:22.975627 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:22.975702 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:22.975820 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.976256 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.976984 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:22.977164 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:22.977236 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:22.977409 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:22.977511 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:22.977594 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:22.977645 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:55:22.977661 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:22.979367 24381 raft_consensus.cc:1261] T 6fa49be235364a568e2bdab5a025f617 P b4c32ce06b5f4f6f9db6e4671d8fde53 [term 1 FOLLOWER]: Refusing update from remote peer 29687b5c1764483dbe09bf8886562735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:22.979393 24512 raft_consensus.cc:1261] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 1 FOLLOWER]: Refusing update from remote peer 29687b5c1764483dbe09bf8886562735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:22.979678 24709 consensus_queue.cc:1035] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "178d35819e794826b82c561b9d269b16" member_type: VOTER last_known_addr { host: "127.5.15.130" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:22.979802 24713 consensus_queue.cc:1035] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c32ce06b5f4f6f9db6e4671d8fde53" member_type: VOTER last_known_addr { host: "127.5.15.129" port: 36291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:22.982188 24718 mvcc.cc:205] Tried to move back new op lower bound from 7333619806121398272 to 7333619806072025088. Current Snapshot: MvccSnapshot[applied={T|T < 7333619806121398272}]
I20260926 14:55:22.982483 24717 mvcc.cc:205] Tried to move back new op lower bound from 7333619806121398272 to 7333619806072025088. Current Snapshot: MvccSnapshot[applied={T|T < 7333619806121398272}]
I20260926 14:55:22.984652 24237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:22.984737 24237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:55:22.985463 24237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14a91a4eed204e6295c6815ead9c87f6: Sending DeleteTablet for 3 replicas of tablet 6fa49be235364a568e2bdab5a025f617
I20260926 14:55:22.985800 24623 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa49be235364a568e2bdab5a025f617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:22 UTC) from {username='slave'} at 127.0.0.1:38550
I20260926 14:55:22.985795 24361 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa49be235364a568e2bdab5a025f617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:22 UTC) from {username='slave'} at 127.0.0.1:40350
I20260926 14:55:22.985795 24492 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fa49be235364a568e2bdab5a025f617 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:55:22 UTC) from {username='slave'} at 127.0.0.1:41028
I20260926 14:55:22.986001 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24299
I20260926 14:55:22.986137 24726 tablet_replica.cc:330] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: stopping tablet replica
I20260926 14:55:22.986224 24726 raft_consensus.cc:2229] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:55:22.986312 24726 raft_consensus.cc:2258] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:22.986477 24727 tablet_replica.cc:330] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: stopping tablet replica
I20260926 14:55:22.986555 24727 raft_consensus.cc:2229] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:55:22.986667 24727 raft_consensus.cc:2258] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:55:22.987174 24726 ts_tablet_manager.cc:1905] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:22.987993 24727 ts_tablet_manager.cc:1905] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:55:22.990207 24726 ts_tablet_manager.cc:1918] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:22.990283 24726 log.cc:1196] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-1/wal/wals/6fa49be235364a568e2bdab5a025f617
I20260926 14:55:22.990535 24726 ts_tablet_manager.cc:1939] T 6fa49be235364a568e2bdab5a025f617 P 178d35819e794826b82c561b9d269b16: Deleting consensus metadata
I20260926 14:55:22.991766 24727 ts_tablet_manager.cc:1918] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:55:22.991850 24727 log.cc:1196] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Deleting WAL directory at /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434201368912-5182-0/minicluster-data/ts-2/wal/wals/6fa49be235364a568e2bdab5a025f617
I20260926 14:55:22.992067 24727 ts_tablet_manager.cc:1939] T 6fa49be235364a568e2bdab5a025f617 P 29687b5c1764483dbe09bf8886562735: Deleting consensus metadata
I20260926 14:55:22.992491 24224 catalog_manager.cc:4841] TS 178d35819e794826b82c561b9d269b16 (127.5.15.130:36907): tablet 6fa49be235364a568e2bdab5a025f617 (table test-table [id=f5939cd33b63410398411af7c2d5af3a]) successfully deleted
I20260926 14:55:22.992868 24224 catalog_manager.cc:4841] TS 29687b5c1764483dbe09bf8886562735 (127.5.15.131:43747): tablet 6fa49be235364a568e2bdab5a025f617 (table test-table [id=f5939cd33b63410398411af7c2d5af3a]) successfully deleted
W20260926 14:55:22.993947 24223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.15.129:36291 (error 108)
I20260926 14:55:22.994143 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24430
I20260926 14:55:22.994751 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:22.994294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.129:36291 (local address 127.0.0.1:40354)
0926 14:55:22.994675 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:22.994717 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.129:36291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
W20260926 14:55:22.994879 24223 catalog_manager.cc:4584] TS b4c32ce06b5f4f6f9db6e4671d8fde53 (127.5.15.129:36291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fa49be235364a568e2bdab5a025f617: Network error: Client connection negotiation failed: client connection to 127.5.15.129:36291: connect: Connection refused (error 111)
I20260926 14:55:22.999418 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24561
I20260926 14:55:23.004091 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24208
2026-09-26T14:55:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (739 ms)
[----------] 6 tests from EncryptionPolicyTest (8021 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:23 dist-test-slave-s4p1 krb5kdc[24732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:23 dist-test-slave-s4p1 krb5kdc[24732](info): set up 2 sockets
Sep 26 14:55:23 dist-test-slave-s4p1 krb5kdc[24732](info): commencing operation
krb5kdc: starting...
W20260926 14:55:24.022571 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:24 dist-test-slave-s4p1 krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:24.133450 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:24.212990 24745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:24.213217 24745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:24.213254 24745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:24.214394 24745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:24.214459 24745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:24.214481 24745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:24.214504 24745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:24.214531 24745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:24.215977 24745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:24.216244 24745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:24.216451 24745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:24.221084 24751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:24.221184 24745 server_base.cc:1034] running on GCE node
W20260926 14:55:24.221076 24753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:24.221076 24750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:24.221590 24745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:24.221863 24745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:24.222999 24745 hybrid_clock.cc:648] HybridClock initialized: now 1790434524222985 us; error 28 us; skew 500 ppm
Sep 26 14:55:24 dist-test-slave-s4p1 krb5kdc[24732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:24.225390 24745 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:24.225826 24745 webserver.cc:460] Webserver started at http://127.0.0.1:37899/ using document root <none> and password file <none>
I20260926 14:55:24.226032 24745 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:24.226095 24745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:24.226255 24745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:24.227391 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "3fe1b4ef28fd42738410470a9a1a6ebc"
format_stamp: "Formatted at 2026-09-26 14:55:24 on dist-test-slave-s4p1"
server_key: "ff8c7d141e91908471d3316a7c67e121"
server_key_iv: "d8d59bf568cc2ad5306ff7a54c47f006"
server_key_version: "encryptionkey@0"
I20260926 14:55:24.227741 24745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "3fe1b4ef28fd42738410470a9a1a6ebc"
format_stamp: "Formatted at 2026-09-26 14:55:24 on dist-test-slave-s4p1"
server_key: "ff8c7d141e91908471d3316a7c67e121"
server_key_iv: "d8d59bf568cc2ad5306ff7a54c47f006"
server_key_version: "encryptionkey@0"
I20260926 14:55:24.229033 24745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:24.229689 24760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:24.229866 24745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:24.229964 24745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "3fe1b4ef28fd42738410470a9a1a6ebc"
format_stamp: "Formatted at 2026-09-26 14:55:24 on dist-test-slave-s4p1"
server_key: "ff8c7d141e91908471d3316a7c67e121"
server_key_iv: "d8d59bf568cc2ad5306ff7a54c47f006"
server_key_version: "encryptionkey@0"
I20260926 14:55:24.230041 24745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:24.259464 24745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:24.261621 24745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:24.261793 24745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:24.265882 24745 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32903
I20260926 14:55:24.265929 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32903 every 8 connection(s)
I20260926 14:55:24.266253 24745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:24.267194 24813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:24.267500 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24745
I20260926 14:55:24.267603 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:24.267791 5182 external_mini_cluster.cc:1428] Setting key d5a6573e34bbbaae5bf91b40564dcb0b
I20260926 14:55:24.269987 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc: Bootstrap starting.
I20260926 14:55:24.270557 24813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:24.270922 24813 log.cc:824] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:24.271651 24813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc: No bootstrap required, opened a new log
I20260926 14:55:24.274026 24813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260926 14:55:24.274149 24813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:24.274171 24813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe1b4ef28fd42738410470a9a1a6ebc, State: Initialized, Role: FOLLOWER
Sep 26 14:55:24 dist-test-slave-s4p1 krb5kdc[24732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:55:24.274322 24813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260926 14:55:24.274427 24813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:24.274470 24813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:24.274520 24813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:24.275252 24813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260926 14:55:24.275364 24813 leader_election.cc:302] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fe1b4ef28fd42738410470a9a1a6ebc; no voters:
I20260926 14:55:24.275517 24813 leader_election.cc:288] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:24.275563 24818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:24.275704 24818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [term 1 LEADER]: Becoming Leader. State: Replica: 3fe1b4ef28fd42738410470a9a1a6ebc, State: Running, Role: LEADER
I20260926 14:55:24.275756 24813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:24.275861 24818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } }
I20260926 14:55:24.276193 24819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fe1b4ef28fd42738410470a9a1a6ebc. Latest consensus state: current_term: 1 leader_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260926 14:55:24.276288 24819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:24.276526 24825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:24.276853 24820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe1b4ef28fd42738410470a9a1a6ebc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32903 } } }
I20260926 14:55:24.277014 24820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:24.277007 24825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:24.278529 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:24.268291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47379 (local address 127.0.0.1:32903)
0926 14:55:24.268447 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:24.268453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:24.268475 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:55:24.268850 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:24.268855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:24.268897 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:24.269005 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:24.269787 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:24.270707 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:24.271728 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:24.272113 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:24.272199 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:24.274803 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:24.274807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:24.274807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:24.274818 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:24.276695 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:24.277214 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:24.277217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:24.277217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:24.277313 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:24.277582 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:24.277583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:24.277583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:24.277916 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:24.278073 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:24.278238 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 14:55:24.278333 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260926 14:55:24.279104 24825 catalog_manager.cc:1347] Generated new cluster ID: 45a2124593b9411d87cfd6e5b73c889b
I20260926 14:55:24.279160 24825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:24.279589 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:24.285344 24825 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:24.285809 24825 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:24.292670 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:24.283057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34641 (local address 127.0.0.1:32903)
0926 14:55:24.283091 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:24.283093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:24.285006 (+ 1913us) server_negotiation.cc:407] Connection header received
0926 14:55:24.285226 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:24.285227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:24.285237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:24.285284 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:24.286515 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:24.287238 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:24.288199 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:24.288493 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:24.288547 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:24.289920 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:24.289921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:24.289922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:24.289928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:24.291478 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:24.291886 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:24.291887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:24.291888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:24.291940 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:24.292154 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:24.292155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:24.292155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:24.292272 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:24.292355 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:24.292530 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:55:24.292588 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:55:24.293794 24825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fe1b4ef28fd42738410470a9a1a6ebc: Generated new TSK 0
I20260926 14:55:24.293947 24825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:24.300065 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24745
W20260926 14:55:24.303784 24730 connection.cc:535] client connection to 127.0.0.1:32903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32903): Connection reset by peer (error 104)
2026-09-26T14:55:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:24 dist-test-slave-s4p1 krb5kdc[24844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:24 dist-test-slave-s4p1 krb5kdc[24844](info): set up 2 sockets
Sep 26 14:55:24 dist-test-slave-s4p1 krb5kdc[24844](info): commencing operation
krb5kdc: starting...
W20260926 14:55:25.330880 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:25 dist-test-slave-s4p1 krb5kdc[24844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:25.441900 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:25.521507 24857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:25.521741 24857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:25.521777 24857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:25.522959 24857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:25.523023 24857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:25.523047 24857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:25.523067 24857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:25.523097 24857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:25.524564 24857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:25.524821 24857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:25.525050 24857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:25.529996 24863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:25.529986 24862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:25.529982 24865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:25.530085 24857 server_base.cc:1034] running on GCE node
I20260926 14:55:25.530445 24857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:25.530735 24857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:25.531926 24857 hybrid_clock.cc:648] HybridClock initialized: now 1790434525531911 us; error 31 us; skew 500 ppm
Sep 26 14:55:25 dist-test-slave-s4p1 krb5kdc[24844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:25.534402 24857 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:25.534935 24857 webserver.cc:460] Webserver started at http://127.0.0.1:45405/ using document root <none> and password file <none>
I20260926 14:55:25.535212 24857 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:25.535256 24857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:25.535418 24857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:25.536571 24857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "1b07a67d20f441afb360ac0544227390"
format_stamp: "Formatted at 2026-09-26 14:55:25 on dist-test-slave-s4p1"
server_key: "f31facda1ee69ed1b70b82cafaa68981"
server_key_iv: "6550b45396531b7b168c7603bc5fe479"
server_key_version: "encryptionkey@0"
I20260926 14:55:25.536926 24857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "1b07a67d20f441afb360ac0544227390"
format_stamp: "Formatted at 2026-09-26 14:55:25 on dist-test-slave-s4p1"
server_key: "f31facda1ee69ed1b70b82cafaa68981"
server_key_iv: "6550b45396531b7b168c7603bc5fe479"
server_key_version: "encryptionkey@0"
I20260926 14:55:25.538187 24857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:25.538924 24872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:25.539086 24857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:25.539175 24857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "1b07a67d20f441afb360ac0544227390"
format_stamp: "Formatted at 2026-09-26 14:55:25 on dist-test-slave-s4p1"
server_key: "f31facda1ee69ed1b70b82cafaa68981"
server_key_iv: "6550b45396531b7b168c7603bc5fe479"
server_key_version: "encryptionkey@0"
I20260926 14:55:25.539255 24857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:25.554253 24857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:25.556471 24857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:25.556628 24857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:25.560685 24857 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46169
I20260926 14:55:25.560734 24924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46169 every 8 connection(s)
I20260926 14:55:25.561074 24857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:25.561987 24925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:25.564163 24925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390: Bootstrap starting.
I20260926 14:55:25.564723 24925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:25.565026 24925 log.cc:824] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:25.565707 24925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390: No bootstrap required, opened a new log
I20260926 14:55:25.566521 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24857
I20260926 14:55:25.566634 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:25.566839 5182 external_mini_cluster.cc:1428] Setting key d93586f034ccb4fb9d21a8e0d08ca3ab
I20260926 14:55:25.567924 24925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b07a67d20f441afb360ac0544227390" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260926 14:55:25.568115 24925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:25.568156 24925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b07a67d20f441afb360ac0544227390, State: Initialized, Role: FOLLOWER
I20260926 14:55:25.568300 24925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b07a67d20f441afb360ac0544227390" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260926 14:55:25.568390 24925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:25.568423 24925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:25.568478 24925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:25.569202 24925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b07a67d20f441afb360ac0544227390" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260926 14:55:25.569307 24925 leader_election.cc:302] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b07a67d20f441afb360ac0544227390; no voters:
I20260926 14:55:25.569459 24925 leader_election.cc:288] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:25.569496 24930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:25.569638 24930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [term 1 LEADER]: Becoming Leader. State: Replica: 1b07a67d20f441afb360ac0544227390, State: Running, Role: LEADER
I20260926 14:55:25.569739 24925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:25.569767 24930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b07a67d20f441afb360ac0544227390" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } }
I20260926 14:55:25.570161 24931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b07a67d20f441afb360ac0544227390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b07a67d20f441afb360ac0544227390" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260926 14:55:25.570168 24932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b07a67d20f441afb360ac0544227390. Latest consensus state: current_term: 1 leader_uuid: "1b07a67d20f441afb360ac0544227390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b07a67d20f441afb360ac0544227390" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46169 } } }
I20260926 14:55:25.570250 24931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:25.570251 24932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:25.571096 24937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:25.571734 24937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:25.573484 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:25.567357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52439 (local address 127.0.0.1:46169)
0926 14:55:25.567543 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:25.567559 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:25.567583 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:55:25.567898 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:25.567904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:25.567948 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:25.568059 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:25.568331 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:25.570108 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:25.571365 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:25.571896 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:25.572004 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:25.572755 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:25.572758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:25.572758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:25.572968 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:25.573004 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:25.573019 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:25.573106 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:25.573229 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 14:55:25.573259 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260926 14:55:25.573501 24937 catalog_manager.cc:1347] Generated new cluster ID: e9565c54dd6b41658facea9a31df2326
I20260926 14:55:25.573571 24937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:25.574476 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:25.578569 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:25.575917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53423 (local address 127.0.0.1:46169)
0926 14:55:25.575952 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:25.575954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:25.575988 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:55:25.576058 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:25.576058 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:25.576068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:25.576121 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:25.576243 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:25.576686 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:25.577389 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:25.577609 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:25.577693 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:25.578346 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:25.578347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:25.578348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:25.578352 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:25.578365 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:25.578369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:25.578422 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:25.578482 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:55:25.578493 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 14:55:25.582648 24937 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:25.583173 24937 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:25.588128 24937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b07a67d20f441afb360ac0544227390: Generated new TSK 0
I20260926 14:55:25.588302 24937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:25.591530 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24857
W20260926 14:55:25.597127 24842 connection.cc:535] client connection to 127.0.0.1:46169 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46169): Connection reset by peer (error 104)
2026-09-26T14:55:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:25 dist-test-slave-s4p1 krb5kdc[24956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:25 dist-test-slave-s4p1 krb5kdc[24956](info): set up 2 sockets
Sep 26 14:55:25 dist-test-slave-s4p1 krb5kdc[24956](info): commencing operation
krb5kdc: starting...
W20260926 14:55:26.614681 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:26 dist-test-slave-s4p1 krb5kdc[24956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:26.727632 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:26.807168 24969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:26.807359 24969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:26.807389 24969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:26.808524 24969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:26.808562 24969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:26.808578 24969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:26.808627 24969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:26.808653 24969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:26.810184 24969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:26.810434 24969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:26.810662 24969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:26.815456 24977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:26.815511 24974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:26.815511 24975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:26.815623 24969 server_base.cc:1034] running on GCE node
I20260926 14:55:26.815904 24969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:26.816187 24969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:26.817329 24969 hybrid_clock.cc:648] HybridClock initialized: now 1790434526817317 us; error 25 us; skew 500 ppm
Sep 26 14:55:26 dist-test-slave-s4p1 krb5kdc[24956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:26.819967 24969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:26.820472 24969 webserver.cc:460] Webserver started at http://127.0.0.1:38895/ using document root <none> and password file <none>
I20260926 14:55:26.820739 24969 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:26.820801 24969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:26.820932 24969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:26.821943 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "2208de6080d3498b812356d88bb3a339"
format_stamp: "Formatted at 2026-09-26 14:55:26 on dist-test-slave-s4p1"
server_key: "02284a251ca5920fed5e1dae6b87fc14"
server_key_iv: "81f84a0f556e7975237d0665b70463e6"
server_key_version: "encryptionkey@0"
I20260926 14:55:26.822283 24969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "2208de6080d3498b812356d88bb3a339"
format_stamp: "Formatted at 2026-09-26 14:55:26 on dist-test-slave-s4p1"
server_key: "02284a251ca5920fed5e1dae6b87fc14"
server_key_iv: "81f84a0f556e7975237d0665b70463e6"
server_key_version: "encryptionkey@0"
I20260926 14:55:26.823597 24969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:26.824318 24984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:26.824486 24969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:26.824571 24969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "2208de6080d3498b812356d88bb3a339"
format_stamp: "Formatted at 2026-09-26 14:55:26 on dist-test-slave-s4p1"
server_key: "02284a251ca5920fed5e1dae6b87fc14"
server_key_iv: "81f84a0f556e7975237d0665b70463e6"
server_key_version: "encryptionkey@0"
I20260926 14:55:26.824622 24969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:26.827725 24969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:26.829800 24969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:26.829900 24969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:26.833456 24969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46827
I20260926 14:55:26.833483 25036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46827 every 8 connection(s)
I20260926 14:55:26.833794 24969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:26.834717 25037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:26.836860 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339: Bootstrap starting.
I20260926 14:55:26.837405 25037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:26.837697 25037 log.cc:824] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:26.838376 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339: No bootstrap required, opened a new log
I20260926 14:55:26.840034 25037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2208de6080d3498b812356d88bb3a339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 14:55:26.840180 25037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:26.840204 25037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2208de6080d3498b812356d88bb3a339, State: Initialized, Role: FOLLOWER
I20260926 14:55:26.840349 25037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2208de6080d3498b812356d88bb3a339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 14:55:26.840435 25037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:26.840477 25037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:26.840525 25037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:26.841022 25037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2208de6080d3498b812356d88bb3a339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 14:55:26.841130 25037 leader_election.cc:302] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2208de6080d3498b812356d88bb3a339; no voters:
I20260926 14:55:26.841305 25037 leader_election.cc:288] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:26.841358 25040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:26.841518 25040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [term 1 LEADER]: Becoming Leader. State: Replica: 2208de6080d3498b812356d88bb3a339, State: Running, Role: LEADER
I20260926 14:55:26.841572 25037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:26.841804 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 24969
I20260926 14:55:26.841917 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:26.842092 5182 external_mini_cluster.cc:1428] Setting key 2802600f368fb825c774378441add63e
I20260926 14:55:26.841629 25040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2208de6080d3498b812356d88bb3a339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } }
I20260926 14:55:26.842821 25041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2208de6080d3498b812356d88bb3a339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2208de6080d3498b812356d88bb3a339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260926 14:55:26.843091 25041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:26.842964 25042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2208de6080d3498b812356d88bb3a339. Latest consensus state: current_term: 1 leader_uuid: "2208de6080d3498b812356d88bb3a339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2208de6080d3498b812356d88bb3a339" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46827 } } }
I20260926 14:55:26.843250 25042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339 [sys.catalog]: This master's current role is: LEADER
W20260926 14:55:26.843360 25056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:26.843402 25056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:26.843631 25058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:26.843794 25058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:26.844604 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:26.842668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41753 (local address 127.0.0.1:46827)
0926 14:55:26.842849 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:26.842858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:26.842889 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:55:26.843084 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:26.843092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:26.843122 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:26.843261 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:26.844082 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:26.844086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:26.844086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:26.844090 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:26.844130 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:26.844144 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:26.844220 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:26.844289 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:55:26.844411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260926 14:55:26.845229 25058 catalog_manager.cc:1347] Generated new cluster ID: 446e6efe7af54f8387e6d30953e8a268
I20260926 14:55:26.845276 25058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:26.845633 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:26.851804 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:26.847066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45919 (local address 127.0.0.1:46827)
0926 14:55:26.847106 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:26.847108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:26.850947 (+ 3839us) server_negotiation.cc:407] Connection header received
0926 14:55:26.851010 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:26.851011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:26.851016 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:26.851075 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:26.851597 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:26.851598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:26.851598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:26.851602 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:26.851616 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:26.851620 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:26.851677 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:26.851730 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 14:55:26.851741 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:55:26.859349 25058 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:26.859506 25058 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:26.879797 25058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2208de6080d3498b812356d88bb3a339: Generated new TSK 0
I20260926 14:55:26.879989 25058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:26.883885 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 24969
2026-09-26T14:55:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:26 dist-test-slave-s4p1 krb5kdc[25068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:26 dist-test-slave-s4p1 krb5kdc[25068](info): set up 2 sockets
Sep 26 14:55:26 dist-test-slave-s4p1 krb5kdc[25068](info): commencing operation
krb5kdc: starting...
W20260926 14:55:27.898641 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:27 dist-test-slave-s4p1 krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:28.009572 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:28.087829 25081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:28.088052 25081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:28.088088 25081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:28.089176 25081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:28.089244 25081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:28.089269 25081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:28.089298 25081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:28.089329 25081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:28.090808 25081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:28.091087 25081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:28.091295 25081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:28.095865 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:28.096026 25081 server_base.cc:1034] running on GCE node
W20260926 14:55:28.095885 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:28.095938 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:28.096364 25081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:28.096658 25081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:28.097826 25081 hybrid_clock.cc:648] HybridClock initialized: now 1790434528097803 us; error 53 us; skew 500 ppm
Sep 26 14:55:28 dist-test-slave-s4p1 krb5kdc[25068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:28.100265 25081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:28.100718 25081 webserver.cc:460] Webserver started at http://127.0.0.1:43545/ using document root <none> and password file <none>
I20260926 14:55:28.100896 25081 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:28.100936 25081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:28.101022 25081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:28.101971 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "b427bfcb66be41a1bc710bf9560d8f81"
format_stamp: "Formatted at 2026-09-26 14:55:28 on dist-test-slave-s4p1"
server_key: "f287c1cc31b59002d9ebc7fceaa81a44"
server_key_iv: "66521454525f3d418e5882917e3da1de"
server_key_version: "encryptionkey@0"
I20260926 14:55:28.102437 25081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "b427bfcb66be41a1bc710bf9560d8f81"
format_stamp: "Formatted at 2026-09-26 14:55:28 on dist-test-slave-s4p1"
server_key: "f287c1cc31b59002d9ebc7fceaa81a44"
server_key_iv: "66521454525f3d418e5882917e3da1de"
server_key_version: "encryptionkey@0"
I20260926 14:55:28.103804 25081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:28.104476 25096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:28.104645 25081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:28.104732 25081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b427bfcb66be41a1bc710bf9560d8f81"
format_stamp: "Formatted at 2026-09-26 14:55:28 on dist-test-slave-s4p1"
server_key: "f287c1cc31b59002d9ebc7fceaa81a44"
server_key_iv: "66521454525f3d418e5882917e3da1de"
server_key_version: "encryptionkey@0"
I20260926 14:55:28.104803 25081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:28.117576 25081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:28.119493 25081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:28.119637 25081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:28.123616 25081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34135
I20260926 14:55:28.123651 25148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34135 every 8 connection(s)
I20260926 14:55:28.123970 25081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:28.124862 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:28.127071 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81: Bootstrap starting.
I20260926 14:55:28.127614 25149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:28.127892 25149 log.cc:824] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:28.128531 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81: No bootstrap required, opened a new log
I20260926 14:55:28.130040 25149 raft_consensus.cc:348] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b427bfcb66be41a1bc710bf9560d8f81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260926 14:55:28.130169 25149 raft_consensus.cc:374] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:28.130218 25149 raft_consensus.cc:729] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b427bfcb66be41a1bc710bf9560d8f81, State: Initialized, Role: FOLLOWER
I20260926 14:55:28.130366 25149 consensus_queue.cc:260] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b427bfcb66be41a1bc710bf9560d8f81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260926 14:55:28.130427 25149 raft_consensus.cc:388] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:28.130447 25149 raft_consensus.cc:482] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:28.130509 25149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:28.131107 25149 raft_consensus.cc:504] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b427bfcb66be41a1bc710bf9560d8f81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260926 14:55:28.131212 25149 leader_election.cc:302] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b427bfcb66be41a1bc710bf9560d8f81; no voters:
I20260926 14:55:28.131388 25149 leader_election.cc:288] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:28.131415 25152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:28.131582 25152 raft_consensus.cc:686] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [term 1 LEADER]: Becoming Leader. State: Replica: b427bfcb66be41a1bc710bf9560d8f81, State: Running, Role: LEADER
I20260926 14:55:28.131649 25149 sys_catalog.cc:564] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:28.131724 25152 consensus_queue.cc:237] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b427bfcb66be41a1bc710bf9560d8f81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } }
I20260926 14:55:28.132073 25153 sys_catalog.cc:455] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b427bfcb66be41a1bc710bf9560d8f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b427bfcb66be41a1bc710bf9560d8f81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } } }
I20260926 14:55:28.132171 25153 sys_catalog.cc:458] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:28.132074 25154 sys_catalog.cc:455] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b427bfcb66be41a1bc710bf9560d8f81. Latest consensus state: current_term: 1 leader_uuid: "b427bfcb66be41a1bc710bf9560d8f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b427bfcb66be41a1bc710bf9560d8f81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34135 } } }
I20260926 14:55:28.132308 25154 sys_catalog.cc:458] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:28.132454 25156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:28.132961 25156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:28.133855 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25081
I20260926 14:55:28.133961 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:28.134127 5182 external_mini_cluster.cc:1428] Setting key d8adebe61b9fba28f3c1edd6c082306e
I20260926 14:55:28.134719 25156 catalog_manager.cc:1347] Generated new cluster ID: fc8e8f1360a74e08a9c278e2c5e32306
I20260926 14:55:28.134832 25156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:55:28 dist-test-slave-s4p1 krb5kdc[25068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:55:28.143525 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:28.134574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54499 (local address 127.0.0.1:34135)
0926 14:55:28.134829 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:28.134839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:28.134870 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:55:28.134985 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:28.134990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:28.135031 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:28.135161 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:28.135773 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:28.136665 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:28.137421 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:28.137839 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:28.137951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:28.140289 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:28.140293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:28.140293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:28.140316 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:28.141965 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:28.142330 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:28.142333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:28.142334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:28.142429 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:28.142664 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:28.142665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:28.142666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:28.142967 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:28.143136 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:28.143202 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 14:55:28.143361 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":73,"threads_started":1}
I20260926 14:55:28.144364 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:28.151678 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:28.145838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50293 (local address 127.0.0.1:34135)
0926 14:55:28.145882 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:28.145884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:28.145964 (+ 80us) server_negotiation.cc:407] Connection header received
0926 14:55:28.146027 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:28.146028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:28.146051 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:28.146091 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:28.146589 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:28.147135 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:28.147813 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:28.148017 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:28.148089 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:28.149164 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:28.149165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:28.149166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:28.149172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:28.150522 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:28.150936 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:28.150937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:28.150937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:28.150987 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:28.151191 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:28.151192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:28.151193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:28.151303 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:28.151403 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:28.151507 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 14:55:28.151579 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:28.176100 25156 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:28.176631 25156 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:28.182580 25156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b427bfcb66be41a1bc710bf9560d8f81: Generated new TSK 0
I20260926 14:55:28.182799 25156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:28.188465 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25081
2026-09-26T14:55:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:28 dist-test-slave-s4p1 krb5kdc[25180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:28 dist-test-slave-s4p1 krb5kdc[25180](info): set up 2 sockets
Sep 26 14:55:28 dist-test-slave-s4p1 krb5kdc[25180](info): commencing operation
krb5kdc: starting...
W20260926 14:55:29.218619 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:29 dist-test-slave-s4p1 krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:29.329164 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:29.408315 25193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:29.408507 25193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:29.408529 25193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:29.409660 25193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:29.409703 25193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:29.409718 25193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:29.409754 25193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:29.409783 25193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:29.411283 25193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:29.411556 25193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:29.411785 25193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:29.416675 25201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:29.416783 25198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:29.416706 25199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:29.416864 25193 server_base.cc:1034] running on GCE node
I20260926 14:55:29.417073 25193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:29.417357 25193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:29.418502 25193 hybrid_clock.cc:648] HybridClock initialized: now 1790434529418481 us; error 37 us; skew 500 ppm
Sep 26 14:55:29 dist-test-slave-s4p1 krb5kdc[25180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:29.421332 25193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:29.421792 25193 webserver.cc:460] Webserver started at http://127.0.0.1:43535/ using document root <none> and password file <none>
I20260926 14:55:29.421981 25193 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:29.422020 25193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:29.422102 25193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:29.423121 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "788e0de22fe54f4fa505b6151794c82d"
format_stamp: "Formatted at 2026-09-26 14:55:29 on dist-test-slave-s4p1"
server_key: "39b6571e83a94cad72bc21d987370a56"
server_key_iv: "147e2c8f5ed1ed88a41b16aba22a6ba4"
server_key_version: "encryptionkey@0"
I20260926 14:55:29.423424 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "788e0de22fe54f4fa505b6151794c82d"
format_stamp: "Formatted at 2026-09-26 14:55:29 on dist-test-slave-s4p1"
server_key: "39b6571e83a94cad72bc21d987370a56"
server_key_iv: "147e2c8f5ed1ed88a41b16aba22a6ba4"
server_key_version: "encryptionkey@0"
I20260926 14:55:29.424594 25193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:29.425237 25208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:29.425407 25193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:29.425486 25193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "788e0de22fe54f4fa505b6151794c82d"
format_stamp: "Formatted at 2026-09-26 14:55:29 on dist-test-slave-s4p1"
server_key: "39b6571e83a94cad72bc21d987370a56"
server_key_iv: "147e2c8f5ed1ed88a41b16aba22a6ba4"
server_key_version: "encryptionkey@0"
I20260926 14:55:29.425537 25193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:29.441821 25193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:29.444097 25193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:29.444209 25193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:29.447952 25193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35901
I20260926 14:55:29.448004 25260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35901 every 8 connection(s)
I20260926 14:55:29.448338 25193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:29.449303 25261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:29.451547 25261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d: Bootstrap starting.
I20260926 14:55:29.452111 25261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:29.452396 25261 log.cc:824] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:29.453074 25261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d: No bootstrap required, opened a new log
I20260926 14:55:29.453379 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25193
I20260926 14:55:29.453485 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:29.453667 5182 external_mini_cluster.cc:1428] Setting key 139c7d34a983668758960bf3ad1d207c
I20260926 14:55:29.455364 25261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e0de22fe54f4fa505b6151794c82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35901 } }
I20260926 14:55:29.455538 25261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:29.455574 25261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788e0de22fe54f4fa505b6151794c82d, State: Initialized, Role: FOLLOWER
I20260926 14:55:29.455739 25261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e0de22fe54f4fa505b6151794c82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35901 } }
I20260926 14:55:29.455845 25261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:29.455899 25261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:29.455950 25261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:29.456458 25261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e0de22fe54f4fa505b6151794c82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35901 } }
I20260926 14:55:29.456576 25261 leader_election.cc:302] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788e0de22fe54f4fa505b6151794c82d; no voters:
I20260926 14:55:29.456728 25261 leader_election.cc:288] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:29.456760 25266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:29.456866 25266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [term 1 LEADER]: Becoming Leader. State: Replica: 788e0de22fe54f4fa505b6151794c82d, State: Running, Role: LEADER
I20260926 14:55:29.456993 25261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:29.457001 25266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e0de22fe54f4fa505b6151794c82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35901 } }
I20260926 14:55:29.457427 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788e0de22fe54f4fa505b6151794c82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e0de22fe54f4fa505b6151794c82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35901 } } }
I20260926 14:55:29.457525 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:29.457437 25268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788e0de22fe54f4fa505b6151794c82d. Latest consensus state: current_term: 1 leader_uuid: "788e0de22fe54f4fa505b6151794c82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788e0de22fe54f4fa505b6151794c82d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35901 } } }
I20260926 14:55:29.457652 25268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d [sys.catalog]: This master's current role is: LEADER
W20260926 14:55:29.458906 25281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:29.459000 25281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:29.459085 25274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:29.459232 25274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:29.460634 25274 catalog_manager.cc:1347] Generated new cluster ID: 9d043fed4ed7456688fd9723a79c1da0
I20260926 14:55:29.460687 25274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:29.460740 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:29.454139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44951 (local address 127.0.0.1:35901)
0926 14:55:29.454309 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:29.454318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:29.454354 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:55:29.455328 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:29.455336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:29.455407 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:29.455554 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:29.455790 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:29.457242 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:29.458459 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:29.459111 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:29.459219 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:29.459936 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:29.459940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:29.459957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:29.460186 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:29.460226 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:29.460239 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:29.460321 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:29.460402 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:55:29.460467 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260926 14:55:29.461689 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:29.465732 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:29.463044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38951 (local address 127.0.0.1:35901)
0926 14:55:29.463081 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:29.463082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:29.463233 (+ 151us) server_negotiation.cc:407] Connection header received
0926 14:55:29.463286 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:29.463286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:29.463297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:29.463350 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:29.463455 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:29.463897 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:29.464598 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:29.464801 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:29.464885 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:29.465518 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:29.465519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:29.465520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:29.465524 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:29.465537 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:29.465541 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:29.465582 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:29.465636 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 14:55:29.465658 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:55:29.486434 25274 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:29.487020 25274 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:29.492225 25274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788e0de22fe54f4fa505b6151794c82d: Generated new TSK 0
I20260926 14:55:29.492415 25274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:29.497378 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25193
2026-09-26T14:55:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:29 dist-test-slave-s4p1 krb5kdc[25292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:29 dist-test-slave-s4p1 krb5kdc[25292](info): set up 2 sockets
Sep 26 14:55:29 dist-test-slave-s4p1 krb5kdc[25292](info): commencing operation
krb5kdc: starting...
W20260926 14:55:30.526551 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:30 dist-test-slave-s4p1 krb5kdc[25292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:30.637197 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:30.714668 25305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:30.714931 25305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:30.714957 25305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:30.716104 25305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:30.716145 25305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:30.716161 25305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:30.716208 25305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:30.716228 25305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:30.717669 25305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:30.717900 25305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:30.718114 25305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:30.722788 25311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:30.722853 25310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:30.722833 25313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:30.723016 25305 server_base.cc:1034] running on GCE node
I20260926 14:55:30.723228 25305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:30.723506 25305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:30.724668 25305 hybrid_clock.cc:648] HybridClock initialized: now 1790434530724656 us; error 38 us; skew 500 ppm
Sep 26 14:55:30 dist-test-slave-s4p1 krb5kdc[25292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:30.730545 25305 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:30.731091 25305 webserver.cc:460] Webserver started at http://127.0.0.1:41549/ using document root <none> and password file <none>
I20260926 14:55:30.731350 25305 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:30.731420 25305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:30.731571 25305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:30.732623 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "b7960ace43ff4c3e8bfef2e725c0f149"
format_stamp: "Formatted at 2026-09-26 14:55:30 on dist-test-slave-s4p1"
server_key: "83138426d375f22e2028b927dda9de37"
server_key_iv: "41093e06e78cb8618feec387eb92b61b"
server_key_version: "encryptionkey@0"
I20260926 14:55:30.732993 25305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "b7960ace43ff4c3e8bfef2e725c0f149"
format_stamp: "Formatted at 2026-09-26 14:55:30 on dist-test-slave-s4p1"
server_key: "83138426d375f22e2028b927dda9de37"
server_key_iv: "41093e06e78cb8618feec387eb92b61b"
server_key_version: "encryptionkey@0"
I20260926 14:55:30.734350 25305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:30.735110 25320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:30.735284 25305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:30.735384 25305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b7960ace43ff4c3e8bfef2e725c0f149"
format_stamp: "Formatted at 2026-09-26 14:55:30 on dist-test-slave-s4p1"
server_key: "83138426d375f22e2028b927dda9de37"
server_key_iv: "41093e06e78cb8618feec387eb92b61b"
server_key_version: "encryptionkey@0"
I20260926 14:55:30.735469 25305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:30.738658 25305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:30.740447 25305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:30.740602 25305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:30.744529 25305 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41225
I20260926 14:55:30.744560 25372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41225 every 8 connection(s)
I20260926 14:55:30.744865 25305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:30.745759 25373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:30.747910 25373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149: Bootstrap starting.
I20260926 14:55:30.748405 25373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:30.748625 25373 log.cc:824] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:30.749248 25373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149: No bootstrap required, opened a new log
I20260926 14:55:30.750733 25373 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 14:55:30.750926 25373 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:30.750950 25373 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7960ace43ff4c3e8bfef2e725c0f149, State: Initialized, Role: FOLLOWER
I20260926 14:55:30.751073 25373 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 14:55:30.751147 25373 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:30.751199 25373 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:30.751264 25373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:30.751332 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25305
I20260926 14:55:30.751446 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:30.751636 5182 external_mini_cluster.cc:1428] Setting key a939ae0cf95fd8040a02930df783f41d
I20260926 14:55:30.751973 25373 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 14:55:30.752110 25373 leader_election.cc:302] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7960ace43ff4c3e8bfef2e725c0f149; no voters:
I20260926 14:55:30.752279 25373 leader_election.cc:288] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:30.752310 25378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:30.752481 25378 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [term 1 LEADER]: Becoming Leader. State: Replica: b7960ace43ff4c3e8bfef2e725c0f149, State: Running, Role: LEADER
I20260926 14:55:30.752550 25373 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:30.752638 25378 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } }
I20260926 14:55:30.753083 25379 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } } }
I20260926 14:55:30.753165 25379 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:30.753361 25380 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7960ace43ff4c3e8bfef2e725c0f149. Latest consensus state: current_term: 1 leader_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7960ace43ff4c3e8bfef2e725c0f149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41225 } } }
I20260926 14:55:30.753520 25380 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:30.753901 25383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:30.753981 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:30.752111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50009 (local address 127.0.0.1:41225)
0926 14:55:30.752288 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:30.752297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:30.752329 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:55:30.752444 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:30.752452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:30.752484 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:30.752885 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:30.753365 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:30.753368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:30.753369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:30.753372 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:30.753409 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:30.753425 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:30.753709 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:30.753743 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 14:55:30.753782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260926 14:55:30.754397 25383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:30.755441 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:30.755930 25383 catalog_manager.cc:1347] Generated new cluster ID: 453c49825632418789be65843e0af961
I20260926 14:55:30.755987 25383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:30.757973 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:30.756924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42105 (local address 127.0.0.1:41225)
0926 14:55:30.756955 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:30.756957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:30.757084 (+ 127us) server_negotiation.cc:407] Connection header received
0926 14:55:30.757138 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:30.757139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:30.757144 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:30.757207 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:30.757760 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:30.757761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:30.757762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:30.757766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:30.757780 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:30.757785 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:30.757846 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:30.757905 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 14:55:30.757918 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:55:30.773536 25383 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:30.773713 25383 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:30.776568 25383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7960ace43ff4c3e8bfef2e725c0f149: Generated new TSK 0
I20260926 14:55:30.776736 25383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:30.785446 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25305
2026-09-26T14:55:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:30 dist-test-slave-s4p1 krb5kdc[25404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:30 dist-test-slave-s4p1 krb5kdc[25404](info): set up 2 sockets
Sep 26 14:55:30 dist-test-slave-s4p1 krb5kdc[25404](info): commencing operation
krb5kdc: starting...
W20260926 14:55:31.802682 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:31 dist-test-slave-s4p1 krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:31.913393 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:31.990602 25417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:31.990865 25417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:31.990906 25417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:31.992038 25417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:31.992105 25417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:31.992131 25417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:31.992167 25417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:31.992190 25417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:31.993587 25417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:31.993837 25417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:31.994041 25417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:55:31.998601 25417 server_base.cc:1034] running on GCE node
W20260926 14:55:31.998648 25422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:31.998734 25425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:31.998788 25423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:31.999078 25417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:31.999434 25417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:32.000597 25417 hybrid_clock.cc:648] HybridClock initialized: now 1790434532000570 us; error 56 us; skew 500 ppm
Sep 26 14:55:31 dist-test-slave-s4p1 krb5kdc[25404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:32.002998 25417 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:32.003490 25417 webserver.cc:460] Webserver started at http://127.0.0.1:39057/ using document root <none> and password file <none>
I20260926 14:55:32.003721 25417 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:32.003791 25417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:32.003944 25417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:32.004928 25417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "4bcca468d37643f293d007d268d3cf2f"
format_stamp: "Formatted at 2026-09-26 14:55:31 on dist-test-slave-s4p1"
server_key: "61172dd813f4f02503ce0796aa694e1a"
server_key_iv: "0a59fe42fffbf46292035a5e99e5653b"
server_key_version: "encryptionkey@0"
I20260926 14:55:32.005290 25417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "4bcca468d37643f293d007d268d3cf2f"
format_stamp: "Formatted at 2026-09-26 14:55:31 on dist-test-slave-s4p1"
server_key: "61172dd813f4f02503ce0796aa694e1a"
server_key_iv: "0a59fe42fffbf46292035a5e99e5653b"
server_key_version: "encryptionkey@0"
I20260926 14:55:32.006510 25417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:32.007288 25432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:32.007459 25417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:32.007552 25417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "4bcca468d37643f293d007d268d3cf2f"
format_stamp: "Formatted at 2026-09-26 14:55:31 on dist-test-slave-s4p1"
server_key: "61172dd813f4f02503ce0796aa694e1a"
server_key_iv: "0a59fe42fffbf46292035a5e99e5653b"
server_key_version: "encryptionkey@0"
I20260926 14:55:32.007642 25417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:32.015947 25417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:32.017755 25417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:32.017917 25417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:32.021744 25417 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41295
I20260926 14:55:32.021773 25484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41295 every 8 connection(s)
I20260926 14:55:32.022074 25417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:32.022987 25485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:32.024948 25485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f: Bootstrap starting.
I20260926 14:55:32.025420 25485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:32.025638 25485 log.cc:824] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:32.026310 25485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f: No bootstrap required, opened a new log
I20260926 14:55:32.027133 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25417
I20260926 14:55:32.027256 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:32.027427 5182 external_mini_cluster.cc:1428] Setting key 4b3d07f239deda0f29e42dbc80436430
I20260926 14:55:32.028357 25485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcca468d37643f293d007d268d3cf2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41295 } }
I20260926 14:55:32.028518 25485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:32.028573 25485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcca468d37643f293d007d268d3cf2f, State: Initialized, Role: FOLLOWER
I20260926 14:55:32.028716 25485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcca468d37643f293d007d268d3cf2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41295 } }
I20260926 14:55:32.028796 25485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:32.028818 25485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:32.028887 25485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:32.029917 25485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcca468d37643f293d007d268d3cf2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41295 } }
I20260926 14:55:32.030019 25485 leader_election.cc:302] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bcca468d37643f293d007d268d3cf2f; no voters:
I20260926 14:55:32.030158 25485 leader_election.cc:288] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:32.030211 25490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:32.030377 25485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:32.030396 25490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [term 1 LEADER]: Becoming Leader. State: Replica: 4bcca468d37643f293d007d268d3cf2f, State: Running, Role: LEADER
I20260926 14:55:32.030481 25490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcca468d37643f293d007d268d3cf2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41295 } }
I20260926 14:55:32.030857 25490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bcca468d37643f293d007d268d3cf2f. Latest consensus state: current_term: 1 leader_uuid: "4bcca468d37643f293d007d268d3cf2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcca468d37643f293d007d268d3cf2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41295 } } }
I20260926 14:55:32.030972 25490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:32.031200 25499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:32.031915 25499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:32.032097 25491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bcca468d37643f293d007d268d3cf2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcca468d37643f293d007d268d3cf2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41295 } } }
I20260926 14:55:32.032230 25491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:32.033591 25499 catalog_manager.cc:1347] Generated new cluster ID: ac3cae3e3dd8462887d18942941037ec
I20260926 14:55:32.033640 25499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:55:32 dist-test-slave-s4p1 krb5kdc[25404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:55:32.039027 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:32.027911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:34651 (local address 127.0.0.1:41295)
0926 14:55:32.028095 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:32.028104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:32.028139 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:55:32.028317 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:32.028324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:32.028395 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:32.028527 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:32.029277 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:32.030703 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:32.031812 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:32.032370 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:32.032470 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:32.035222 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:32.035226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:32.035226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:32.035238 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:32.037222 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:32.037675 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:32.037678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:32.037679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:32.037810 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:32.038050 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:32.038051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:32.038051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:32.038368 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:32.038544 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:32.038627 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:55:32.038767 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260926 14:55:32.040072 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:32.049394 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:32.041437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:51257 (local address 127.0.0.1:41295)
0926 14:55:32.041506 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:32.041507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:32.041646 (+ 139us) server_negotiation.cc:407] Connection header received
0926 14:55:32.041676 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:32.041676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:32.041686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:32.041750 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:32.042287 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:32.043115 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:32.044053 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:32.044377 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:32.044447 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:32.045744 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:32.045745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:32.045746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:32.045753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:32.047685 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:32.048351 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:32.048352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:32.048352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:32.048432 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:32.048758 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:32.048759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:32.048760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:32.048900 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:32.048991 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:32.049248 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 14:55:32.049313 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:55:32.054387 25499 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:32.055145 25499 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:32.063035 25499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bcca468d37643f293d007d268d3cf2f: Generated new TSK 0
I20260926 14:55:32.063200 25499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:32.066453 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25417
W20260926 14:55:32.070611 25402 connection.cc:535] client connection to 127.0.0.1:41295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41295): Connection reset by peer (error 104)
2026-09-26T14:55:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:32 dist-test-slave-s4p1 krb5kdc[25516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:32 dist-test-slave-s4p1 krb5kdc[25516](info): set up 2 sockets
Sep 26 14:55:32 dist-test-slave-s4p1 krb5kdc[25516](info): commencing operation
krb5kdc: starting...
W20260926 14:55:33.090703 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:33 dist-test-slave-s4p1 krb5kdc[25516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:33.201066 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:33.277882 25529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:33.278069 25529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:33.278090 25529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:33.279529 25529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:33.279585 25529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:33.279609 25529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:33.279628 25529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:33.279649 25529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:33.281737 25529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:33.281931 25529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:33.282167 25529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:33.286976 25535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:33.287006 25534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:33.287006 25537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:33.287173 25529 server_base.cc:1034] running on GCE node
I20260926 14:55:33.287407 25529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:33.287722 25529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:33.288885 25529 hybrid_clock.cc:648] HybridClock initialized: now 1790434533288871 us; error 25 us; skew 500 ppm
Sep 26 14:55:33 dist-test-slave-s4p1 krb5kdc[25516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:33.291276 25529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:33.291734 25529 webserver.cc:460] Webserver started at http://127.0.0.1:42235/ using document root <none> and password file <none>
I20260926 14:55:33.291970 25529 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:33.292039 25529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:33.292168 25529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:33.293118 25529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "467b14c7c90c4c70b331e43920b3a074"
format_stamp: "Formatted at 2026-09-26 14:55:33 on dist-test-slave-s4p1"
server_key: "c7a25c08901350ddbc7ada18a3532881"
server_key_iv: "8d6c69016866b9b6e2e4aeffa1414688"
server_key_version: "encryptionkey@0"
I20260926 14:55:33.293467 25529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "467b14c7c90c4c70b331e43920b3a074"
format_stamp: "Formatted at 2026-09-26 14:55:33 on dist-test-slave-s4p1"
server_key: "c7a25c08901350ddbc7ada18a3532881"
server_key_iv: "8d6c69016866b9b6e2e4aeffa1414688"
server_key_version: "encryptionkey@0"
I20260926 14:55:33.294806 25529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:33.295531 25544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:33.295693 25529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:33.295794 25529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "467b14c7c90c4c70b331e43920b3a074"
format_stamp: "Formatted at 2026-09-26 14:55:33 on dist-test-slave-s4p1"
server_key: "c7a25c08901350ddbc7ada18a3532881"
server_key_iv: "8d6c69016866b9b6e2e4aeffa1414688"
server_key_version: "encryptionkey@0"
I20260926 14:55:33.295878 25529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:33.305889 25529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:33.307734 25529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:33.307880 25529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:33.311753 25529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45183
I20260926 14:55:33.311785 25596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45183 every 8 connection(s)
I20260926 14:55:33.312098 25529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:33.312954 25597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:33.315054 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25529
I20260926 14:55:33.315106 25597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074: Bootstrap starting.
I20260926 14:55:33.315171 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:33.315336 5182 external_mini_cluster.cc:1428] Setting key ed887622ba397af79650f032897902ab
I20260926 14:55:33.315708 25597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:33.316073 25597 log.cc:824] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:33.316764 25597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074: No bootstrap required, opened a new log
I20260926 14:55:33.318819 25597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467b14c7c90c4c70b331e43920b3a074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260926 14:55:33.318992 25597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:33.319038 25597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467b14c7c90c4c70b331e43920b3a074, State: Initialized, Role: FOLLOWER
I20260926 14:55:33.319157 25597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467b14c7c90c4c70b331e43920b3a074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260926 14:55:33.319249 25597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:33.319298 25597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:33.319358 25597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:33.320037 25597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467b14c7c90c4c70b331e43920b3a074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260926 14:55:33.320159 25597 leader_election.cc:302] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467b14c7c90c4c70b331e43920b3a074; no voters:
I20260926 14:55:33.320312 25597 leader_election.cc:288] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:33.320511 25602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:33.320608 25597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:33.320684 25602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [term 1 LEADER]: Becoming Leader. State: Replica: 467b14c7c90c4c70b331e43920b3a074, State: Running, Role: LEADER
I20260926 14:55:33.320741 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:33.315785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:45773 (local address 127.0.0.1:45183)
0926 14:55:33.315958 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:33.315964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:33.315987 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:55:33.316412 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:33.316418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:33.316461 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:33.316566 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:33.316769 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:33.317637 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:33.318838 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:33.319336 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:33.319459 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:33.320002 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:33.320005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:33.320005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:33.320181 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:33.320215 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:33.320234 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:33.320327 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:33.320480 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:55:33.320508 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260926 14:55:33.320763 25602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467b14c7c90c4c70b331e43920b3a074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } }
I20260926 14:55:33.321486 25603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467b14c7c90c4c70b331e43920b3a074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467b14c7c90c4c70b331e43920b3a074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } } }
I20260926 14:55:33.321560 25603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:33.321674 25603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467b14c7c90c4c70b331e43920b3a074. Latest consensus state: current_term: 1 leader_uuid: "467b14c7c90c4c70b331e43920b3a074" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467b14c7c90c4c70b331e43920b3a074" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45183 } } }
I20260926 14:55:33.321729 25603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:33.321768 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 14:55:33.322408 25617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:33.322465 25617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:33.322515 25611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:33.322669 25611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:33.324173 25611 catalog_manager.cc:1347] Generated new cluster ID: 2f09c8a5a3644180ad54cb144e727b5f
I20260926 14:55:33.324225 25611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:33.326778 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:33.323398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:48623 (local address 127.0.0.1:45183)
0926 14:55:33.323436 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:33.323438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:33.323551 (+ 113us) server_negotiation.cc:407] Connection header received
0926 14:55:33.323617 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:33.323618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:33.323627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:33.323682 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:33.323801 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:33.324516 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:33.325516 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:33.325829 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:33.325890 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:33.326496 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:33.326497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:33.326497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:33.326501 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:33.326514 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:33.326517 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:33.326573 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:33.326664 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:55:33.326672 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:33.330229 25611 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:33.330981 25611 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:33.340857 25611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467b14c7c90c4c70b331e43920b3a074: Generated new TSK 0
I20260926 14:55:33.341039 25611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:33.345125 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25529
W20260926 14:55:33.349267 25514 connection.cc:535] client connection to 127.0.0.1:45183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45183): Connection reset by peer (error 104)
2026-09-26T14:55:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:33 dist-test-slave-s4p1 krb5kdc[25628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:33 dist-test-slave-s4p1 krb5kdc[25628](info): set up 2 sockets
Sep 26 14:55:33 dist-test-slave-s4p1 krb5kdc[25628](info): commencing operation
krb5kdc: starting...
W20260926 14:55:34.386833 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:34 dist-test-slave-s4p1 krb5kdc[25628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:34.503067 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:34.580585 25641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:34.580816 25641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:34.580856 25641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:34.581943 25641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:34.582021 25641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:34.582046 25641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:34.582072 25641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:34.582101 25641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:34.583535 25641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:34.583804 25641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:34.584021 25641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:34.588688 25646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:34.588665 25647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:34.588665 25649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:34.588771 25641 server_base.cc:1034] running on GCE node
I20260926 14:55:34.589095 25641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:34.589367 25641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:34.590512 25641 hybrid_clock.cc:648] HybridClock initialized: now 1790434534590504 us; error 29 us; skew 500 ppm
Sep 26 14:55:34 dist-test-slave-s4p1 krb5kdc[25628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:34.599718 25641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:34.600155 25641 webserver.cc:460] Webserver started at http://127.0.0.1:44019/ using document root <none> and password file <none>
I20260926 14:55:34.600346 25641 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:34.600385 25641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:34.600493 25641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:34.601475 25641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60"
format_stamp: "Formatted at 2026-09-26 14:55:34 on dist-test-slave-s4p1"
server_key: "4b53448869098f866dad512120100f4f"
server_key_iv: "717979084da6bca1fac737a0b1c05e8f"
server_key_version: "encryptionkey@0"
I20260926 14:55:34.601779 25641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60"
format_stamp: "Formatted at 2026-09-26 14:55:34 on dist-test-slave-s4p1"
server_key: "4b53448869098f866dad512120100f4f"
server_key_iv: "717979084da6bca1fac737a0b1c05e8f"
server_key_version: "encryptionkey@0"
I20260926 14:55:34.603050 25641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:34.603713 25656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:34.603897 25641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:34.603971 25641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60"
format_stamp: "Formatted at 2026-09-26 14:55:34 on dist-test-slave-s4p1"
server_key: "4b53448869098f866dad512120100f4f"
server_key_iv: "717979084da6bca1fac737a0b1c05e8f"
server_key_version: "encryptionkey@0"
I20260926 14:55:34.604025 25641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:34.607154 25641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:34.608819 25641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:34.608925 25641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:34.612504 25641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40421
I20260926 14:55:34.612567 25708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40421 every 8 connection(s)
I20260926 14:55:34.612870 25641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:34.613763 25709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:34.615967 25709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60: Bootstrap starting.
I20260926 14:55:34.616480 25709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:34.616765 25709 log.cc:824] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:34.617471 25709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60: No bootstrap required, opened a new log
I20260926 14:55:34.617614 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25641
I20260926 14:55:34.617725 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:34.617928 5182 external_mini_cluster.cc:1428] Setting key 61796ea24323a5ac47877b0b0a3a2565
I20260926 14:55:34.619782 25709 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260926 14:55:34.619969 25709 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:34.620002 25709 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c0f51bb5dc4d0cbcc0da5fc2b41d60, State: Initialized, Role: FOLLOWER
I20260926 14:55:34.620121 25709 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260926 14:55:34.620203 25709 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:34.620245 25709 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:34.620296 25709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:34.620980 25709 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260926 14:55:34.621081 25709 leader_election.cc:302] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c0f51bb5dc4d0cbcc0da5fc2b41d60; no voters:
I20260926 14:55:34.621171 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:34.618424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:43343 (local address 127.0.0.1:40421)
0926 14:55:34.618612 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:34.618622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:34.618654 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:55:34.619979 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:34.619988 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:34.620021 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:34.620173 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:34.620776 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:34.620779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:34.620780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:34.620784 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:34.620824 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:34.620838 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:34.620914 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:34.620956 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:55:34.620997 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260926 14:55:34.621248 25709 leader_election.cc:288] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:34.621345 25714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:34.621479 25709 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:34.621488 25714 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [term 1 LEADER]: Becoming Leader. State: Replica: d5c0f51bb5dc4d0cbcc0da5fc2b41d60, State: Running, Role: LEADER
I20260926 14:55:34.621594 25714 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } }
I20260926 14:55:34.621925 25714 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c0f51bb5dc4d0cbcc0da5fc2b41d60. Latest consensus state: current_term: 1 leader_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } } }
I20260926 14:55:34.622005 25714 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:34.622175 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:34.622184 25716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c0f51bb5dc4d0cbcc0da5fc2b41d60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40421 } } }
I20260926 14:55:34.622303 25716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:34.622848 25723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:34.624213 25723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:34.624802 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:34.623648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:59789 (local address 127.0.0.1:40421)
0926 14:55:34.623698 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:34.623700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:34.623828 (+ 128us) server_negotiation.cc:407] Connection header received
0926 14:55:34.623904 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:34.623904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:34.623908 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:34.623956 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:34.624556 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:34.624557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:34.624558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:34.624562 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:34.624576 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:34.624581 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:34.624639 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:34.624730 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 14:55:34.624744 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:55:34.625782 25723 catalog_manager.cc:1347] Generated new cluster ID: 6d83aa6608124470916a0a6f8e218472
I20260926 14:55:34.625829 25723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:34.638420 25723 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:34.638572 25723 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:34.657071 25723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c0f51bb5dc4d0cbcc0da5fc2b41d60: Generated new TSK 0
I20260926 14:55:34.657285 25723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:34.665047 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25641
2026-09-26T14:55:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:34 dist-test-slave-s4p1 krb5kdc[25740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:34 dist-test-slave-s4p1 krb5kdc[25740](info): set up 2 sockets
Sep 26 14:55:34 dist-test-slave-s4p1 krb5kdc[25740](info): commencing operation
krb5kdc: starting...
W20260926 14:55:35.698818 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:35 dist-test-slave-s4p1 krb5kdc[25740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:35.809955 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:35.889107 25753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:35.889356 25753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:35.889395 25753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:35.890528 25753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:35.890595 25753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:35.890618 25753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:35.890640 25753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:35.890668 25753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:35.892156 25753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:35.892429 25753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:35.892652 25753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:35.897421 25761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:35.897472 25759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:35.897471 25758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:35.897614 25753 server_base.cc:1034] running on GCE node
I20260926 14:55:35.897888 25753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:35.898216 25753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:35.899374 25753 hybrid_clock.cc:648] HybridClock initialized: now 1790434535899360 us; error 46 us; skew 500 ppm
Sep 26 14:55:35 dist-test-slave-s4p1 krb5kdc[25740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:35.905618 25753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:35.906049 25753 webserver.cc:460] Webserver started at http://127.0.0.1:39311/ using document root <none> and password file <none>
I20260926 14:55:35.906288 25753 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:35.906332 25753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:35.906498 25753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:35.907577 25753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "a4b95970be244eb79635b861a4a5798e"
format_stamp: "Formatted at 2026-09-26 14:55:35 on dist-test-slave-s4p1"
server_key: "961af4bae018efcb568195d2d4514d80"
server_key_iv: "bff6251cee2a947bb8d397066ecc3387"
server_key_version: "encryptionkey@0"
I20260926 14:55:35.907922 25753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "a4b95970be244eb79635b861a4a5798e"
format_stamp: "Formatted at 2026-09-26 14:55:35 on dist-test-slave-s4p1"
server_key: "961af4bae018efcb568195d2d4514d80"
server_key_iv: "bff6251cee2a947bb8d397066ecc3387"
server_key_version: "encryptionkey@0"
I20260926 14:55:35.909165 25753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:35.909864 25768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:35.910069 25753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:35.910152 25753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "a4b95970be244eb79635b861a4a5798e"
format_stamp: "Formatted at 2026-09-26 14:55:35 on dist-test-slave-s4p1"
server_key: "961af4bae018efcb568195d2d4514d80"
server_key_iv: "bff6251cee2a947bb8d397066ecc3387"
server_key_version: "encryptionkey@0"
I20260926 14:55:35.910244 25753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:35.918686 25753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:35.920629 25753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:35.920778 25753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:35.924726 25753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40203
I20260926 14:55:35.924765 25820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40203 every 8 connection(s)
I20260926 14:55:35.925071 25753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:35.925992 25821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:35.928107 25821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e: Bootstrap starting.
I20260926 14:55:35.928609 25821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:35.928840 25821 log.cc:824] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:35.929435 25821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e: No bootstrap required, opened a new log
I20260926 14:55:35.931051 25821 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b95970be244eb79635b861a4a5798e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40203 } }
I20260926 14:55:35.931176 25821 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:35.931198 25821 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b95970be244eb79635b861a4a5798e, State: Initialized, Role: FOLLOWER
I20260926 14:55:35.931321 25821 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b95970be244eb79635b861a4a5798e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40203 } }
I20260926 14:55:35.931398 25821 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:35.931420 25821 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:35.931456 25821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:35.931922 25821 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b95970be244eb79635b861a4a5798e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40203 } }
I20260926 14:55:35.931998 25821 leader_election.cc:302] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4b95970be244eb79635b861a4a5798e; no voters:
I20260926 14:55:35.932107 25821 leader_election.cc:288] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:35.932176 25824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:35.932322 25824 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [term 1 LEADER]: Becoming Leader. State: Replica: a4b95970be244eb79635b861a4a5798e, State: Running, Role: LEADER
I20260926 14:55:35.932353 25821 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:35.932440 25824 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b95970be244eb79635b861a4a5798e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40203 } }
I20260926 14:55:35.932829 25826 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4b95970be244eb79635b861a4a5798e. Latest consensus state: current_term: 1 leader_uuid: "a4b95970be244eb79635b861a4a5798e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b95970be244eb79635b861a4a5798e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40203 } } }
I20260926 14:55:35.932979 25826 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:35.932950 25825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4b95970be244eb79635b861a4a5798e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4b95970be244eb79635b861a4a5798e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40203 } } }
I20260926 14:55:35.933076 25825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:35.933458 25833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:35.933846 25833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:35.934199 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25753
I20260926 14:55:35.934336 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:35.934562 5182 external_mini_cluster.cc:1428] Setting key bc30de90ca32c5e17cabbff8fe7b67aa
I20260926 14:55:35.935653 25833 catalog_manager.cc:1347] Generated new cluster ID: 99fbef23ce5746eebf9456a948345a7e
I20260926 14:55:35.935701 25833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:55:35 dist-test-slave-s4p1 krb5kdc[25740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:55:35.948613 25833 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:35.949275 25833 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:35.950054 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:35.935000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:55695 (local address 127.0.0.1:40203)
0926 14:55:35.938888 (+ 3888us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:35.938897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:35.938931 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:55:35.939042 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:35.939049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:35.939106 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:35.939253 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:35.939954 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:35.941735 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:35.942743 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:35.943378 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:35.943497 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:35.946148 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:35.946151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:35.946151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:35.946162 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:35.948111 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:35.948678 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:35.948701 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:35.948701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:35.948794 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:35.949064 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:35.949064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:35.949065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:35.949327 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:35.949522 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:35.949668 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 14:55:35.949854 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3841,"thread_start_us":129,"threads_started":1}
I20260926 14:55:35.951011 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:35.955250 25833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4b95970be244eb79635b861a4a5798e: Generated new TSK 0
I20260926 14:55:35.955418 25833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:35.961692 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:35.955068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:47333 (local address 127.0.0.1:40203)
0926 14:55:35.955126 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:35.955128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:35.955413 (+ 285us) server_negotiation.cc:407] Connection header received
0926 14:55:35.955433 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:35.955434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:35.955443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:35.955485 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:35.956155 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:35.956621 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:35.957661 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:35.957884 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:35.957937 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:35.959244 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:35.959246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:35.959246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:35.959251 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:35.960587 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:35.960985 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:35.960986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:35.960986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:35.961030 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:35.961246 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:35.961247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:35.961247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:35.961341 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:35.961419 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:35.961572 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:55:35.961616 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:55:35.963429 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25753
W20260926 14:55:35.967243 25738 connection.cc:535] client connection to 127.0.0.1:40203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40203): Connection reset by peer (error 104)
2026-09-26T14:55:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:36 dist-test-slave-s4p1 krb5kdc[25852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:36 dist-test-slave-s4p1 krb5kdc[25852](info): set up 2 sockets
Sep 26 14:55:36 dist-test-slave-s4p1 krb5kdc[25852](info): commencing operation
krb5kdc: starting...
W20260926 14:55:36.990815 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:37 dist-test-slave-s4p1 krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:37.107594 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:37.187502 25865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:37.187701 25865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:37.187726 25865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:37.188843 25865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:37.188884 25865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:37.188896 25865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:37.188910 25865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:37.188920 25865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:37.190377 25865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39353
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:37.190611 25865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:37.190897 25865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:37.195555 25870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:37.195549 25871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:37.195549 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:37.195688 25865 server_base.cc:1034] running on GCE node
I20260926 14:55:37.195961 25865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:37.196254 25865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:37.197412 25865 hybrid_clock.cc:648] HybridClock initialized: now 1790434537197399 us; error 47 us; skew 500 ppm
Sep 26 14:55:37 dist-test-slave-s4p1 krb5kdc[25852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:37.199848 25865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:37.200274 25865 webserver.cc:460] Webserver started at http://127.0.0.1:41903/ using document root <none> and password file <none>
I20260926 14:55:37.200462 25865 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:37.200500 25865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:37.200583 25865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:37.201561 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "2d75a8afdf864f08bf25cafb9cdbe673"
format_stamp: "Formatted at 2026-09-26 14:55:37 on dist-test-slave-s4p1"
server_key: "e829e93ae78f81402fef7ce83f4e590b"
server_key_iv: "fef475c889ec7580879f71c86f501ae0"
server_key_version: "encryptionkey@0"
I20260926 14:55:37.201846 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "2d75a8afdf864f08bf25cafb9cdbe673"
format_stamp: "Formatted at 2026-09-26 14:55:37 on dist-test-slave-s4p1"
server_key: "e829e93ae78f81402fef7ce83f4e590b"
server_key_iv: "fef475c889ec7580879f71c86f501ae0"
server_key_version: "encryptionkey@0"
I20260926 14:55:37.203117 25865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:37.203750 25880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:37.203915 25865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:37.203997 25865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "2d75a8afdf864f08bf25cafb9cdbe673"
format_stamp: "Formatted at 2026-09-26 14:55:37 on dist-test-slave-s4p1"
server_key: "e829e93ae78f81402fef7ce83f4e590b"
server_key_iv: "fef475c889ec7580879f71c86f501ae0"
server_key_version: "encryptionkey@0"
I20260926 14:55:37.204056 25865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:37.213716 25865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:37.215806 25865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:37.215906 25865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:37.219544 25865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39353
I20260926 14:55:37.219597 25932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39353 every 8 connection(s)
I20260926 14:55:37.219907 25865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:37.220750 25933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:37.221802 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25865
I20260926 14:55:37.221927 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:37.222100 5182 external_mini_cluster.cc:1428] Setting key c203c310cda5ab6a05c556c215647321
I20260926 14:55:37.223491 25933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673: Bootstrap starting.
I20260926 14:55:37.224156 25933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:37.224454 25933 log.cc:824] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:37.225188 25933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673: No bootstrap required, opened a new log
I20260926 14:55:37.227401 25933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260926 14:55:37.227574 25933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:37.227615 25933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d75a8afdf864f08bf25cafb9cdbe673, State: Initialized, Role: FOLLOWER
I20260926 14:55:37.227790 25933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260926 14:55:37.227840 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:37.222592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:48499 (local address 127.0.0.1:39353)
0926 14:55:37.222832 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:37.222841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:37.222873 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:55:37.223431 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:37.223440 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:37.223502 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:37.223640 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:37.223982 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:37.224975 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:37.226051 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:37.226444 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:37.226538 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:37.227222 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:37.227225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:37.227225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:37.227374 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:37.227402 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:37.227411 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:37.227473 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:37.227637 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:55:37.227663 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260926 14:55:37.227885 25933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:37.227910 25933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:37.227965 25933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:37.228518 25933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260926 14:55:37.228636 25933 leader_election.cc:302] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d75a8afdf864f08bf25cafb9cdbe673; no voters:
I20260926 14:55:37.228801 25933 leader_election.cc:288] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:37.228825 25938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:37.228868 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:37.228952 25938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [term 1 LEADER]: Becoming Leader. State: Replica: 2d75a8afdf864f08bf25cafb9cdbe673, State: Running, Role: LEADER
I20260926 14:55:37.229049 25933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:37.229085 25938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } }
I20260926 14:55:37.229451 25938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d75a8afdf864f08bf25cafb9cdbe673. Latest consensus state: current_term: 1 leader_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } } }
I20260926 14:55:37.229703 25938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:37.229626 25939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d75a8afdf864f08bf25cafb9cdbe673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39353 } } }
I20260926 14:55:37.230110 25939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:37.230298 25947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:37.230899 25947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:37.232415 25947 catalog_manager.cc:1347] Generated new cluster ID: 65bbecc2285d4794bb19b545f511fa2c
I20260926 14:55:37.232463 25947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:37.234354 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:37.230814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:47867 (local address 127.0.0.1:39353)
0926 14:55:37.230865 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:37.230867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:37.231010 (+ 143us) server_negotiation.cc:407] Connection header received
0926 14:55:37.231090 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:37.231091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:37.231100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:37.231160 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:37.231287 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:37.231993 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:37.233021 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:37.233302 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:37.233343 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:37.234096 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:37.234098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:37.234098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:37.234102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:37.234115 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:37.234119 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:37.234149 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:37.234278 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 14:55:37.234288 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 14:55:37.261487 25947 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:37.262034 25947 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:37.269699 25947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d75a8afdf864f08bf25cafb9cdbe673: Generated new TSK 0
I20260926 14:55:37.269881 25947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:37.278280 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25865
W20260926 14:55:37.282684 25850 connection.cc:535] client connection to 127.0.0.1:39353 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39353): Connection reset by peer (error 104)
2026-09-26T14:55:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:37 dist-test-slave-s4p1 krb5kdc[25964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:37 dist-test-slave-s4p1 krb5kdc[25964](info): set up 2 sockets
Sep 26 14:55:37 dist-test-slave-s4p1 krb5kdc[25964](info): commencing operation
krb5kdc: starting...
W20260926 14:55:38.294947 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:38 dist-test-slave-s4p1 krb5kdc[25964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:55:38.409582 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:38.489262 25977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:38.489492 25977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:38.489530 25977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:38.490688 25977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:38.490757 25977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:38.490809 25977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:38.490841 25977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:38.490868 25977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:38.492342 25977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:38.492609 25977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:38.492856 25977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:38.497469 25982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:38.497489 25985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:38.497469 25983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:38.497709 25977 server_base.cc:1034] running on GCE node
I20260926 14:55:38.497954 25977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:38.498297 25977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:38.499456 25977 hybrid_clock.cc:648] HybridClock initialized: now 1790434538499453 us; error 37 us; skew 500 ppm
Sep 26 14:55:38 dist-test-slave-s4p1 krb5kdc[25964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:38.501964 25977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:55:38.502424 25977 webserver.cc:460] Webserver started at http://127.0.0.1:43287/ using document root <none> and password file <none>
I20260926 14:55:38.502696 25977 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:38.502801 25977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:38.502967 25977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:38.503997 25977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "1330fb3ae7c7416e9f7155234acfa1b5"
format_stamp: "Formatted at 2026-09-26 14:55:38 on dist-test-slave-s4p1"
server_key: "285a267616bbf1f757c54597e483e89e"
server_key_iv: "4c16232b15e3e15a491bfb1d7ceca2b7"
server_key_version: "encryptionkey@0"
I20260926 14:55:38.504348 25977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "1330fb3ae7c7416e9f7155234acfa1b5"
format_stamp: "Formatted at 2026-09-26 14:55:38 on dist-test-slave-s4p1"
server_key: "285a267616bbf1f757c54597e483e89e"
server_key_iv: "4c16232b15e3e15a491bfb1d7ceca2b7"
server_key_version: "encryptionkey@0"
I20260926 14:55:38.505594 25977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:38.506271 25992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:38.506440 25977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:38.506536 25977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "1330fb3ae7c7416e9f7155234acfa1b5"
format_stamp: "Formatted at 2026-09-26 14:55:38 on dist-test-slave-s4p1"
server_key: "285a267616bbf1f757c54597e483e89e"
server_key_iv: "4c16232b15e3e15a491bfb1d7ceca2b7"
server_key_version: "encryptionkey@0"
I20260926 14:55:38.506619 25977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:38.509708 25977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:38.511973 25977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:38.512127 25977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:38.516069 25977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45327
I20260926 14:55:38.516098 26044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45327 every 8 connection(s)
I20260926 14:55:38.516453 25977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:38.517340 26045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:38.519570 26045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5: Bootstrap starting.
I20260926 14:55:38.520107 26045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:38.520375 26045 log.cc:824] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:38.521032 26045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5: No bootstrap required, opened a new log
I20260926 14:55:38.522538 26045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45327 } }
I20260926 14:55:38.522665 26045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:38.522722 26045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1330fb3ae7c7416e9f7155234acfa1b5, State: Initialized, Role: FOLLOWER
I20260926 14:55:38.522917 26045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45327 } }
I20260926 14:55:38.522991 26045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:38.523013 26045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:38.523075 26045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:38.523574 26045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45327 } }
I20260926 14:55:38.523679 26045 leader_election.cc:302] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1330fb3ae7c7416e9f7155234acfa1b5; no voters:
I20260926 14:55:38.523866 26045 leader_election.cc:288] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:38.523895 26048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:38.523937 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 25977
I20260926 14:55:38.524008 26048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [term 1 LEADER]: Becoming Leader. State: Replica: 1330fb3ae7c7416e9f7155234acfa1b5, State: Running, Role: LEADER
I20260926 14:55:38.524047 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:38.524075 26048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45327 } }
I20260926 14:55:38.524107 26045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:38.524217 5182 external_mini_cluster.cc:1428] Setting key 02700c5c3c91dbdd7def6fbdcea9c2b4
I20260926 14:55:38.524358 26048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1330fb3ae7c7416e9f7155234acfa1b5. Latest consensus state: current_term: 1 leader_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45327 } } }
I20260926 14:55:38.524413 26048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:38.524557 26049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1330fb3ae7c7416e9f7155234acfa1b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45327 } } }
I20260926 14:55:38.524664 26049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:38.525385 26053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:38.525907 26053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:38.526984 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:38.524868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:52429 (local address 127.0.0.1:45327)
0926 14:55:38.525231 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:38.525241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:38.525268 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:55:38.525536 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:38.525565 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:38.525598 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:38.525743 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:38.526371 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:38.526375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:38.526375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:38.526379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:38.526435 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:38.526469 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:38.526700 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:38.526737 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 14:55:38.526807 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":67,"threads_started":1}
I20260926 14:55:38.527981 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:38.530191 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:38.529244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:54117 (local address 127.0.0.1:45327)
0926 14:55:38.529276 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:38.529278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:38.529385 (+ 107us) server_negotiation.cc:407] Connection header received
0926 14:55:38.529448 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:38.529449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:38.529462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:38.529507 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:38.529992 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:38.529993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:38.529993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:38.529996 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:38.530006 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:38.530009 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:38.530048 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:38.530100 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 14:55:38.530110 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 14:55:38.531157 26053 catalog_manager.cc:1347] Generated new cluster ID: ed82bb888c92481ba0ce49af4a938c64
I20260926 14:55:38.531219 26053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:38.554548 26053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:38.554783 26053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:38.565976 26053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1330fb3ae7c7416e9f7155234acfa1b5: Generated new TSK 0
I20260926 14:55:38.566171 26053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:38.574607 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 25977
2026-09-26T14:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:38 dist-test-slave-s4p1 krb5kdc[26076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:38 dist-test-slave-s4p1 krb5kdc[26076](info): set up 2 sockets
Sep 26 14:55:38 dist-test-slave-s4p1 krb5kdc[26076](info): commencing operation
krb5kdc: starting...
W20260926 14:55:39.594986 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:39 dist-test-slave-s4p1 krb5kdc[26076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:39.705325 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:39797
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:39797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:39.784941 26089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:39.785153 26089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:39.785188 26089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:39.786314 26089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:39.786379 26089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:39.786402 26089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:39.786424 26089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:39.786454 26089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:39.787842 26089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:39797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:39.788098 26089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:39.788306 26089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:39.792963 26097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:39.793085 26089 server_base.cc:1034] running on GCE node
W20260926 14:55:39.792948 26095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:39.792933 26094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:39.793434 26089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:39.793700 26089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:39.794860 26089 hybrid_clock.cc:648] HybridClock initialized: now 1790434539794846 us; error 44 us; skew 500 ppm
Sep 26 14:55:39 dist-test-slave-s4p1 krb5kdc[26076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434539, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:39.797272 26089 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:39.797714 26089 webserver.cc:460] Webserver started at http://127.5.15.190:42419/ using document root <none> and password file <none>
I20260926 14:55:39.797919 26089 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:39.797982 26089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:39.798125 26089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:39.799193 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb"
format_stamp: "Formatted at 2026-09-26 14:55:39 on dist-test-slave-s4p1"
server_key: "7608389b2003c9c1917a5bd2715e6c2f"
server_key_iv: "d4108e4ab9d5b9c057526b644b6d569a"
server_key_version: "encryptionkey@0"
I20260926 14:55:39.799561 26089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb"
format_stamp: "Formatted at 2026-09-26 14:55:39 on dist-test-slave-s4p1"
server_key: "7608389b2003c9c1917a5bd2715e6c2f"
server_key_iv: "d4108e4ab9d5b9c057526b644b6d569a"
server_key_version: "encryptionkey@0"
I20260926 14:55:39.800822 26089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:39.801528 26104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:39.801692 26089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:39.801795 26089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb"
format_stamp: "Formatted at 2026-09-26 14:55:39 on dist-test-slave-s4p1"
server_key: "7608389b2003c9c1917a5bd2715e6c2f"
server_key_iv: "d4108e4ab9d5b9c057526b644b6d569a"
server_key_version: "encryptionkey@0"
I20260926 14:55:39.801868 26089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:39.825815 26089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:39.828037 26089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:39.828190 26089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:39.832038 26089 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:39797
I20260926 14:55:39.832072 26156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:39797 every 8 connection(s)
I20260926 14:55:39.832419 26089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:39.833187 26157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:39.835422 26157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb: Bootstrap starting.
I20260926 14:55:39.835983 26157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:39.836256 26157 log.cc:824] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:39.836879 26157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb: No bootstrap required, opened a new log
I20260926 14:55:39.838418 26157 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39797 } }
I20260926 14:55:39.838546 26157 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:39.838568 26157 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd89d7e9bb94372a9e1c307ce9d6dcb, State: Initialized, Role: FOLLOWER
I20260926 14:55:39.838740 26157 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39797 } }
I20260926 14:55:39.838853 26157 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:39.838876 26157 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:39.838979 26157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:39.839720 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26089
I20260926 14:55:39.839852 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:39.839912 26157 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39797 } }
I20260926 14:55:39.840020 26157 leader_election.cc:302] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd89d7e9bb94372a9e1c307ce9d6dcb; no voters:
I20260926 14:55:39.840068 5182 external_mini_cluster.cc:1428] Setting key 5c2212b10a29e3ebbb5071f85b744605
I20260926 14:55:39.840178 26157 leader_election.cc:288] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:39.840209 26160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:39.840381 26160 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [term 1 LEADER]: Becoming Leader. State: Replica: cfd89d7e9bb94372a9e1c307ce9d6dcb, State: Running, Role: LEADER
I20260926 14:55:39.840415 26157 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:39.840552 26160 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39797 } }
I20260926 14:55:39.840929 26162 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd89d7e9bb94372a9e1c307ce9d6dcb. Latest consensus state: current_term: 1 leader_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39797 } } }
I20260926 14:55:39.840921 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd89d7e9bb94372a9e1c307ce9d6dcb" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 39797 } } }
I20260926 14:55:39.841013 26162 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:39.841025 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:39.841364 26168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:39.842056 26168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:39.843550 26168 catalog_manager.cc:1347] Generated new cluster ID: 6d9702c04da74aeda8656d7ebaf1a555
I20260926 14:55:39.843600 26168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 14:55:39 dist-test-slave-s4p1 krb5kdc[26076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:39.850847 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:39.840675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41377 (local address 127.5.15.190:39797)
0926 14:55:39.840855 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:39.840864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:39.841075 (+ 211us) server_negotiation.cc:407] Connection header received
0926 14:55:39.841208 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:39.841217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:39.841281 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:39.841422 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:39.842336 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:39.843736 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:39.844474 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:39.844865 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:39.845024 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:39.847321 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:39.847325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:39.847325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:39.847336 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:39.849164 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:39.849613 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:39.849616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:39.849616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:39.849721 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:39.849972 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:39.849973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:39.849973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:39.850236 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:39.850380 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:39.850523 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 14:55:39.850610 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260926 14:55:39.851904 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:39.857724 26168 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:39.858481 26168 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:39.860663 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:39.853367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43023 (local address 127.5.15.190:39797)
0926 14:55:39.853396 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:39.853397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:39.853660 (+ 263us) server_negotiation.cc:407] Connection header received
0926 14:55:39.853712 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:39.853713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:39.853723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:39.853774 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:39.854370 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:39.855068 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:39.856087 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:39.856413 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:39.856473 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:39.857795 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:39.857796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:39.857796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:39.857802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:39.859523 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:39.859914 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:39.859915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:39.859916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:39.859965 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:39.860183 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:39.860184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:39.860184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:39.860288 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:39.860389 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:39.860528 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:55:39.860582 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 14:55:39.863986 26168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd89d7e9bb94372a9e1c307ce9d6dcb: Generated new TSK 0
I20260926 14:55:39.864135 26168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:39.867668 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26089
W20260926 14:55:39.871474 26074 connection.cc:535] client connection to 127.5.15.190:39797 recv error: Network error: failed to read from TLS socket (remote: 127.5.15.190:39797): Connection reset by peer (error 104)
2026-09-26T14:55:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:39 dist-test-slave-s4p1 krb5kdc[26188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:39 dist-test-slave-s4p1 krb5kdc[26188](info): set up 2 sockets
Sep 26 14:55:39 dist-test-slave-s4p1 krb5kdc[26188](info): commencing operation
krb5kdc: starting...
W20260926 14:55:40.878819 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:40 dist-test-slave-s4p1 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:40.989578 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:35039
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:35039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:41.067986 26201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:41.068199 26201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:41.068236 26201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:41.069326 26201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:41.069391 26201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:41.069413 26201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:41.069433 26201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:41.069567 26201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:41.071321 26201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:35039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:35039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:41.071583 26201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:41.071808 26201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:41.076581 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:41.076776 26209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:41.076778 26206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:41.076877 26201 server_base.cc:1034] running on GCE node
I20260926 14:55:41.077198 26201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:41.077481 26201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:41.078619 26201 hybrid_clock.cc:648] HybridClock initialized: now 1790434541078607 us; error 28 us; skew 500 ppm
Sep 26 14:55:41 dist-test-slave-s4p1 krb5kdc[26188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434541, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:41.086148 26201 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:41.086624 26201 webserver.cc:460] Webserver started at http://127.5.15.190:37461/ using document root <none> and password file <none>
I20260926 14:55:41.086907 26201 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:41.086974 26201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:41.087131 26201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:41.088168 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "0578e0e7d9de44929ab17b7caa29f20c"
format_stamp: "Formatted at 2026-09-26 14:55:41 on dist-test-slave-s4p1"
server_key: "af17f09472ba0352fa5a39706f5918ca"
server_key_iv: "5f397aedf31520a1b8832afcb1143a0f"
server_key_version: "encryptionkey@0"
I20260926 14:55:41.088524 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "0578e0e7d9de44929ab17b7caa29f20c"
format_stamp: "Formatted at 2026-09-26 14:55:41 on dist-test-slave-s4p1"
server_key: "af17f09472ba0352fa5a39706f5918ca"
server_key_iv: "5f397aedf31520a1b8832afcb1143a0f"
server_key_version: "encryptionkey@0"
I20260926 14:55:41.089802 26201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 14:55:41.090485 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:41.090651 26201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:41.090749 26201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "0578e0e7d9de44929ab17b7caa29f20c"
format_stamp: "Formatted at 2026-09-26 14:55:41 on dist-test-slave-s4p1"
server_key: "af17f09472ba0352fa5a39706f5918ca"
server_key_iv: "5f397aedf31520a1b8832afcb1143a0f"
server_key_version: "encryptionkey@0"
I20260926 14:55:41.090838 26201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:41.099336 26201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:41.101070 26201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:41.101228 26201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:41.105068 26201 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:35039
I20260926 14:55:41.105109 26268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:35039 every 8 connection(s)
I20260926 14:55:41.105430 26201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:41.106261 26269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:41.108369 26269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c: Bootstrap starting.
I20260926 14:55:41.108837 26269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:41.109056 26269 log.cc:824] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:41.109645 26269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c: No bootstrap required, opened a new log
I20260926 14:55:41.111181 26269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0578e0e7d9de44929ab17b7caa29f20c" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35039 } }
I20260926 14:55:41.111327 26269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:41.111348 26269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0578e0e7d9de44929ab17b7caa29f20c, State: Initialized, Role: FOLLOWER
I20260926 14:55:41.111439 26269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0578e0e7d9de44929ab17b7caa29f20c" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35039 } }
I20260926 14:55:41.111490 26269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:41.111510 26269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:41.111536 26269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:41.112393 26269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0578e0e7d9de44929ab17b7caa29f20c" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35039 } }
I20260926 14:55:41.112478 26269 leader_election.cc:302] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0578e0e7d9de44929ab17b7caa29f20c; no voters:
I20260926 14:55:41.112602 26269 leader_election.cc:288] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:41.112656 26272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:41.112807 26272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [term 1 LEADER]: Becoming Leader. State: Replica: 0578e0e7d9de44929ab17b7caa29f20c, State: Running, Role: LEADER
I20260926 14:55:41.112857 26269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:41.112941 26272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0578e0e7d9de44929ab17b7caa29f20c" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35039 } }
I20260926 14:55:41.113307 26273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0578e0e7d9de44929ab17b7caa29f20c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0578e0e7d9de44929ab17b7caa29f20c" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35039 } } }
I20260926 14:55:41.113334 26274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0578e0e7d9de44929ab17b7caa29f20c. Latest consensus state: current_term: 1 leader_uuid: "0578e0e7d9de44929ab17b7caa29f20c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0578e0e7d9de44929ab17b7caa29f20c" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35039 } } }
I20260926 14:55:41.113471 26274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:41.113963 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26201
I20260926 14:55:41.114085 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:41.114295 5182 external_mini_cluster.cc:1428] Setting key 853ddabe58902978d070135a457332e0
I20260926 14:55:41.114473 26273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c [sys.catalog]: This master's current role is: LEADER
W20260926 14:55:41.114627 26288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:41.114840 26288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:41.114907 26281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:41.115067 26281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:41.116511 26281 catalog_manager.cc:1347] Generated new cluster ID: 0d4d3c5f225c4144a46778f508074f5d
I20260926 14:55:41.116562 26281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:41.119341 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:41.114694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43625 (local address 127.5.15.190:35039)
0926 14:55:41.115002 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:41.115010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:41.115037 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:55:41.115149 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:41.115156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:41.115207 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:41.115316 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:41.115543 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:41.116652 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:41.117660 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:41.118030 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:41.118110 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:41.118686 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:41.118689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:41.118689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:41.118919 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:41.118946 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:41.118956 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:41.119021 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:41.119102 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:55:41.119147 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260926 14:55:41.120157 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:41.124290 26291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:41.121453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43933 (local address 127.5.15.190:35039)
0926 14:55:41.121486 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:41.121488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:41.121605 (+ 117us) server_negotiation.cc:407] Connection header received
0926 14:55:41.121631 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:41.121632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:41.121641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:41.121699 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:41.121783 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:41.122528 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:41.123185 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:41.123458 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:41.123510 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:41.124070 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:41.124071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:41.124072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:41.124075 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:41.124086 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:41.124090 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:41.124164 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:41.124216 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 14:55:41.124225 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 14:55:41.146975 26281 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:41.147746 26281 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:41.158772 26281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0578e0e7d9de44929ab17b7caa29f20c: Generated new TSK 0
I20260926 14:55:41.158972 26281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:41.167419 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26201
W20260926 14:55:41.171730 26186 connection.cc:535] client connection to 127.5.15.190:35039 recv error: Network error: failed to read from TLS socket (remote: 127.5.15.190:35039): Connection reset by peer (error 104)
2026-09-26T14:55:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:41 dist-test-slave-s4p1 krb5kdc[26300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:41 dist-test-slave-s4p1 krb5kdc[26300](info): set up 2 sockets
Sep 26 14:55:41 dist-test-slave-s4p1 krb5kdc[26300](info): commencing operation
krb5kdc: starting...
W20260926 14:55:42.186724 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:42 dist-test-slave-s4p1 krb5kdc[26300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:42.296729 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:35971
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:42.376042 26313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:42.376260 26313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:42.376293 26313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:42.377357 26313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:42.377417 26313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:42.377441 26313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:42.377470 26313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:42.377499 26313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:42.378979 26313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:35971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:42.379254 26313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:42.379467 26313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:42.384075 26318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:42.384064 26321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:42.384141 26319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:42.384204 26313 server_base.cc:1034] running on GCE node
I20260926 14:55:42.384594 26313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:42.384891 26313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:42.386053 26313 hybrid_clock.cc:648] HybridClock initialized: now 1790434542386028 us; error 30 us; skew 500 ppm
Sep 26 14:55:42 dist-test-slave-s4p1 krb5kdc[26300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434542, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:42.388352 26313 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:42.388754 26313 webserver.cc:460] Webserver started at http://127.5.15.190:43175/ using document root <none> and password file <none>
I20260926 14:55:42.388978 26313 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:42.389042 26313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:42.389185 26313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:42.390236 26313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "b7bfcc279f3c43ac9599407ca1475a92"
format_stamp: "Formatted at 2026-09-26 14:55:42 on dist-test-slave-s4p1"
server_key: "08602579a28eda4bd09c339a31e2e1f8"
server_key_iv: "5ed1a74b536856ad32fadab82bb86a9a"
server_key_version: "encryptionkey@0"
I20260926 14:55:42.390612 26313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "b7bfcc279f3c43ac9599407ca1475a92"
format_stamp: "Formatted at 2026-09-26 14:55:42 on dist-test-slave-s4p1"
server_key: "08602579a28eda4bd09c339a31e2e1f8"
server_key_iv: "5ed1a74b536856ad32fadab82bb86a9a"
server_key_version: "encryptionkey@0"
I20260926 14:55:42.391925 26313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 14:55:42.392590 26328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:42.392753 26313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:42.392846 26313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "b7bfcc279f3c43ac9599407ca1475a92"
format_stamp: "Formatted at 2026-09-26 14:55:42 on dist-test-slave-s4p1"
server_key: "08602579a28eda4bd09c339a31e2e1f8"
server_key_iv: "5ed1a74b536856ad32fadab82bb86a9a"
server_key_version: "encryptionkey@0"
I20260926 14:55:42.392949 26313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:42.396168 26313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:42.397846 26313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:42.397995 26313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:42.401773 26313 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:35971
I20260926 14:55:42.401801 26380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:35971 every 8 connection(s)
I20260926 14:55:42.402112 26313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:42.403084 26381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:42.405066 26381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92: Bootstrap starting.
I20260926 14:55:42.405576 26381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:42.405853 26381 log.cc:824] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:42.406464 26381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92: No bootstrap required, opened a new log
I20260926 14:55:42.408002 26381 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bfcc279f3c43ac9599407ca1475a92" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35971 } }
I20260926 14:55:42.408134 26381 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:42.408193 26381 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bfcc279f3c43ac9599407ca1475a92, State: Initialized, Role: FOLLOWER
I20260926 14:55:42.408367 26381 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bfcc279f3c43ac9599407ca1475a92" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35971 } }
I20260926 14:55:42.408442 26381 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:42.408481 26381 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:42.408526 26381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:42.409261 26381 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bfcc279f3c43ac9599407ca1475a92" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35971 } }
I20260926 14:55:42.409390 26381 leader_election.cc:302] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7bfcc279f3c43ac9599407ca1475a92; no voters:
I20260926 14:55:42.409582 26381 leader_election.cc:288] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:42.409621 26384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:42.409804 26384 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [term 1 LEADER]: Becoming Leader. State: Replica: b7bfcc279f3c43ac9599407ca1475a92, State: Running, Role: LEADER
I20260926 14:55:42.409894 26381 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:42.409957 26384 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bfcc279f3c43ac9599407ca1475a92" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35971 } }
I20260926 14:55:42.410280 26385 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7bfcc279f3c43ac9599407ca1475a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bfcc279f3c43ac9599407ca1475a92" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35971 } } }
I20260926 14:55:42.410374 26385 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:42.410290 26386 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7bfcc279f3c43ac9599407ca1475a92. Latest consensus state: current_term: 1 leader_uuid: "b7bfcc279f3c43ac9599407ca1475a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bfcc279f3c43ac9599407ca1475a92" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 35971 } } }
I20260926 14:55:42.410509 26386 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:42.410791 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26313
I20260926 14:55:42.410830 26393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:42.410894 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:42.411060 5182 external_mini_cluster.cc:1428] Setting key 224a0f5388a4f061fab619b01bc8cbd2
I20260926 14:55:42.411252 26393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:42.412914 26393 catalog_manager.cc:1347] Generated new cluster ID: 9da0949c4690464fbdf3495aaf66e507
I20260926 14:55:42.412962 26393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:42.415486 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:42.411732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46893 (local address 127.5.15.190:35971)
0926 14:55:42.411891 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:42.411907 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:42.412274 (+ 367us) server_negotiation.cc:407] Connection header received
0926 14:55:42.412463 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:42.412470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:42.412517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:42.412658 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:42.415042 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:42.415046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:42.415046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:42.415051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:42.415098 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:42.415112 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:42.415179 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:42.415276 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 14:55:42.415311 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260926 14:55:42.419727 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:42.422214 26398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:42.421183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38589 (local address 127.5.15.190:35971)
0926 14:55:42.421217 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:42.421219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:42.421353 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:55:42.421407 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:42.421408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:42.421413 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:42.421464 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:42.422015 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:42.422017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:42.422017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:42.422020 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:42.422034 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:42.422038 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:42.422089 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:42.422130 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:55:42.422141 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:55:42.427455 26393 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:42.427639 26393 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:42.433166 26393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7bfcc279f3c43ac9599407ca1475a92: Generated new TSK 0
I20260926 14:55:42.433337 26393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:42.440764 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26313
2026-09-26T14:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:42 dist-test-slave-s4p1 krb5kdc[26412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:42 dist-test-slave-s4p1 krb5kdc[26412](info): set up 2 sockets
Sep 26 14:55:42 dist-test-slave-s4p1 krb5kdc[26412](info): commencing operation
krb5kdc: starting...
W20260926 14:55:43.454888 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:43 dist-test-slave-s4p1 krb5kdc[26412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:43.564846 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37343
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:37343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:43.643033 26425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:43.643236 26425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:43.643260 26425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:43.644304 26425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:43.644337 26425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:43.644351 26425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:43.644363 26425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:43.644375 26425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:43.645721 26425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:37343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:37343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:43.645895 26425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:43.646047 26425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:43.650607 26433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:43.650590 26431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:43.650662 26425 server_base.cc:1034] running on GCE node
W20260926 14:55:43.650589 26430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:43.651024 26425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:43.651281 26425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:43.652416 26425 hybrid_clock.cc:648] HybridClock initialized: now 1790434543652395 us; error 37 us; skew 500 ppm
Sep 26 14:55:43 dist-test-slave-s4p1 krb5kdc[26412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434543, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:43.654706 26425 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:43.655125 26425 webserver.cc:460] Webserver started at http://127.5.15.190:36227/ using document root <none> and password file <none>
I20260926 14:55:43.655345 26425 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:43.655386 26425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:43.655529 26425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:43.656476 26425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "60a3fa1aaaf740ceb7bc79e604434c95"
format_stamp: "Formatted at 2026-09-26 14:55:43 on dist-test-slave-s4p1"
server_key: "d5fb71d6da6f55f38d70f709ee25aa5c"
server_key_iv: "d0e9dfe86c5b4b00ff283909fcc07fdd"
server_key_version: "encryptionkey@0"
I20260926 14:55:43.656809 26425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "60a3fa1aaaf740ceb7bc79e604434c95"
format_stamp: "Formatted at 2026-09-26 14:55:43 on dist-test-slave-s4p1"
server_key: "d5fb71d6da6f55f38d70f709ee25aa5c"
server_key_iv: "d0e9dfe86c5b4b00ff283909fcc07fdd"
server_key_version: "encryptionkey@0"
I20260926 14:55:43.658017 26425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 14:55:43.658780 26440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:43.658946 26425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:43.659024 26425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "60a3fa1aaaf740ceb7bc79e604434c95"
format_stamp: "Formatted at 2026-09-26 14:55:43 on dist-test-slave-s4p1"
server_key: "d5fb71d6da6f55f38d70f709ee25aa5c"
server_key_iv: "d0e9dfe86c5b4b00ff283909fcc07fdd"
server_key_version: "encryptionkey@0"
I20260926 14:55:43.659073 26425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:43.680687 26425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:43.682915 26425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:43.683019 26425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:43.686525 26425 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:37343
I20260926 14:55:43.686586 26492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:37343 every 8 connection(s)
I20260926 14:55:43.686910 26425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:43.687775 26493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:43.689069 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26425
I20260926 14:55:43.689163 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:43.689378 5182 external_mini_cluster.cc:1428] Setting key ffd15bfcf0457fd9a75add23c40f8076
I20260926 14:55:43.690382 26493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95: Bootstrap starting.
I20260926 14:55:43.691069 26493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:43.691388 26493 log.cc:824] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:43.692158 26493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95: No bootstrap required, opened a new log
I20260926 14:55:43.694303 26493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37343 } }
I20260926 14:55:43.694463 26493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:43.694495 26493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a3fa1aaaf740ceb7bc79e604434c95, State: Initialized, Role: FOLLOWER
I20260926 14:55:43.694618 26493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37343 } }
I20260926 14:55:43.694680 26493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:43.694700 26493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:43.694726 26493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:43.695313 26493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37343 } }
I20260926 14:55:43.695396 26493 leader_election.cc:302] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60a3fa1aaaf740ceb7bc79e604434c95; no voters:
I20260926 14:55:43.695570 26493 leader_election.cc:288] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:43.695593 26498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:43.695699 26498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [term 1 LEADER]: Becoming Leader. State: Replica: 60a3fa1aaaf740ceb7bc79e604434c95, State: Running, Role: LEADER
I20260926 14:55:43.695806 26493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:43.695856 26498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37343 } }
I20260926 14:55:43.696229 26499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37343 } } }
I20260926 14:55:43.696244 26500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60a3fa1aaaf740ceb7bc79e604434c95. Latest consensus state: current_term: 1 leader_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a3fa1aaaf740ceb7bc79e604434c95" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 37343 } } }
I20260926 14:55:43.696327 26499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:43.696338 26500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95 [sys.catalog]: This master's current role is: LEADER
Sep 26 14:55:43 dist-test-slave-s4p1 krb5kdc[26412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
W20260926 14:55:43.697667 26514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:43.697722 26514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:43.697775 26505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:43.697906 26505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:43.699456 26505 catalog_manager.cc:1347] Generated new cluster ID: 6ee5315eec954ab7baae291524e0d423
I20260926 14:55:43.699509 26505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:43.700553 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:43.689864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53877 (local address 127.5.15.190:37343)
0926 14:55:43.690044 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:43.690053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:43.690084 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:55:43.690593 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:43.690601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:43.690662 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:43.690851 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:43.691439 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:43.692820 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:43.693564 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:43.694254 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:43.694366 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:43.696969 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:43.696974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:43.696974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:43.696985 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:43.698939 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:43.699306 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:43.699309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:43.699309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:43.699419 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:43.699630 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:43.699631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:43.699631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:43.699887 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:43.700001 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:43.700235 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 14:55:43.700353 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260926 14:55:43.701787 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:43.709818 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:43.703171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38667 (local address 127.5.15.190:37343)
0926 14:55:43.703199 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:43.703200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:43.703305 (+ 105us) server_negotiation.cc:407] Connection header received
0926 14:55:43.703358 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:43.703358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:43.703366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:43.703421 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:43.704021 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:43.704688 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:43.705655 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:43.705953 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:43.706008 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:43.707300 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:43.707301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:43.707301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:43.707307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:43.708710 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:43.709074 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:43.709075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:43.709075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:43.709128 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:43.709353 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:43.709353 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:43.709354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:43.709454 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:43.709528 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:43.709679 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:55:43.709743 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 14:55:43.712579 26505 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:43.713083 26505 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:43.720592 26505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60a3fa1aaaf740ceb7bc79e604434c95: Generated new TSK 0
I20260926 14:55:43.720755 26505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:43.724623 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26425
2026-09-26T14:55:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:43 dist-test-slave-s4p1 krb5kdc[26524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:43 dist-test-slave-s4p1 krb5kdc[26524](info): set up 2 sockets
Sep 26 14:55:43 dist-test-slave-s4p1 krb5kdc[26524](info): commencing operation
krb5kdc: starting...
W20260926 14:55:44.744549 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:44 dist-test-slave-s4p1 krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:44.857322 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40131
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:40131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:44.933395 26537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:44.933607 26537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:44.933645 26537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:44.934732 26537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:44.934825 26537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:44.934847 26537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:44.934865 26537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:44.934896 26537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:44.936282 26537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:40131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:40131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:44.936545 26537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:44.936760 26537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:44.941457 26543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:44.941443 26545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:44.941442 26542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:44.941833 26537 server_base.cc:1034] running on GCE node
I20260926 14:55:44.942088 26537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:44.942368 26537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:44.943529 26537 hybrid_clock.cc:648] HybridClock initialized: now 1790434544943514 us; error 26 us; skew 500 ppm
Sep 26 14:55:44 dist-test-slave-s4p1 krb5kdc[26524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434544, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:44.945755 26537 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:44.946254 26537 webserver.cc:460] Webserver started at http://127.5.15.190:43691/ using document root <none> and password file <none>
I20260926 14:55:44.946484 26537 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:44.946550 26537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:44.946684 26537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:44.947721 26537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112"
format_stamp: "Formatted at 2026-09-26 14:55:44 on dist-test-slave-s4p1"
server_key: "4ff35f063779cc5e3b8ae24b1b28729c"
server_key_iv: "7c4ab6156928498f32e31106478664df"
server_key_version: "encryptionkey@0"
I20260926 14:55:44.948100 26537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112"
format_stamp: "Formatted at 2026-09-26 14:55:44 on dist-test-slave-s4p1"
server_key: "4ff35f063779cc5e3b8ae24b1b28729c"
server_key_iv: "7c4ab6156928498f32e31106478664df"
server_key_version: "encryptionkey@0"
I20260926 14:55:44.949507 26537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:44.950256 26552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:44.950496 26537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:55:44.950608 26537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112"
format_stamp: "Formatted at 2026-09-26 14:55:44 on dist-test-slave-s4p1"
server_key: "4ff35f063779cc5e3b8ae24b1b28729c"
server_key_iv: "7c4ab6156928498f32e31106478664df"
server_key_version: "encryptionkey@0"
I20260926 14:55:44.950685 26537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:44.961601 26537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:44.963402 26537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:44.963551 26537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:44.967149 26537 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:40131
I20260926 14:55:44.967180 26604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:40131 every 8 connection(s)
I20260926 14:55:44.967489 26537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:44.968222 26605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:44.970285 26605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112: Bootstrap starting.
I20260926 14:55:44.970886 26605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:44.971170 26605 log.cc:824] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:44.971827 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26537
I20260926 14:55:44.971872 26605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112: No bootstrap required, opened a new log
I20260926 14:55:44.971966 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:44.972147 5182 external_mini_cluster.cc:1428] Setting key 65d9752c1d53e67411a0c861310258b6
I20260926 14:55:44.973699 26605 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40131 } }
I20260926 14:55:44.973855 26605 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:44.973891 26605 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cc0d6ca7e54b22b5d488ff9fa58112, State: Initialized, Role: FOLLOWER
I20260926 14:55:44.974069 26605 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40131 } }
I20260926 14:55:44.974176 26605 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:44.974217 26605 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:44.974274 26605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:44.974821 26605 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40131 } }
I20260926 14:55:44.974937 26605 leader_election.cc:302] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6cc0d6ca7e54b22b5d488ff9fa58112; no voters:
I20260926 14:55:44.975083 26605 leader_election.cc:288] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:44.975119 26610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:44.975220 26610 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [term 1 LEADER]: Becoming Leader. State: Replica: c6cc0d6ca7e54b22b5d488ff9fa58112, State: Running, Role: LEADER
I20260926 14:55:44.975329 26605 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:44.975354 26610 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40131 } }
I20260926 14:55:44.975723 26610 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6cc0d6ca7e54b22b5d488ff9fa58112. Latest consensus state: current_term: 1 leader_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40131 } } }
I20260926 14:55:44.975873 26610 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:44.976131 26619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:44.976660 26619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:44.976859 26611 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cc0d6ca7e54b22b5d488ff9fa58112" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 40131 } } }
I20260926 14:55:44.976985 26611 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:44.978456 26619 catalog_manager.cc:1347] Generated new cluster ID: f2affd072ba94646a81b1cea45fccc36
I20260926 14:55:44.978502 26619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:44.978832 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:44.972624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46411 (local address 127.5.15.190:40131)
0926 14:55:44.972792 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:44.972804 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:44.972831 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:55:44.973685 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:44.973690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:44.973733 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:44.973838 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:44.974096 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:44.975481 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:44.976675 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:44.977237 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:44.977356 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:44.978058 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:44.978061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:44.978062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:44.978281 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:44.978317 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:44.978331 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:44.978403 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:44.978516 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 14:55:44.978583 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260926 14:55:44.983409 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:44.987459 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:44.984747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36417 (local address 127.5.15.190:40131)
0926 14:55:44.984781 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:44.984783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:44.984883 (+ 100us) server_negotiation.cc:407] Connection header received
0926 14:55:44.984932 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:44.984933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:44.984942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:44.984998 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:44.985134 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:44.985571 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:44.986274 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:44.986475 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:44.986557 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:44.987222 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:44.987224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:44.987224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:44.987228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:44.987241 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:44.987245 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:44.987298 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:44.987352 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 14:55:44.987372 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:55:45.001332 26619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:45.002019 26619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:45.006382 26619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6cc0d6ca7e54b22b5d488ff9fa58112: Generated new TSK 0
I20260926 14:55:45.006534 26619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:45.008351 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26537
2026-09-26T14:55:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:45 dist-test-slave-s4p1 krb5kdc[26636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:45 dist-test-slave-s4p1 krb5kdc[26636](info): set up 2 sockets
Sep 26 14:55:45 dist-test-slave-s4p1 krb5kdc[26636](info): commencing operation
krb5kdc: starting...
W20260926 14:55:46.014896 5182 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:46 dist-test-slave-s4p1 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:46.125437 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:33501
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:33501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:46.203104 26649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:46.203316 26649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:46.203351 26649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:46.204406 26649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:46.204466 26649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:46.204487 26649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:46.204506 26649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:46.204533 26649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:46.205843 26649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:33501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:33501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:46.206094 26649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:46.206305 26649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:46.210950 26657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:46.210960 26654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:46.210961 26655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:46.211236 26649 server_base.cc:1034] running on GCE node
I20260926 14:55:46.211536 26649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:46.211829 26649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:46.213023 26649 hybrid_clock.cc:648] HybridClock initialized: now 1790434546212999 us; error 39 us; skew 500 ppm
Sep 26 14:55:46 dist-test-slave-s4p1 krb5kdc[26636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434546, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:46.219870 26649 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:46.220278 26649 webserver.cc:460] Webserver started at http://127.5.15.190:35949/ using document root <none> and password file <none>
I20260926 14:55:46.220439 26649 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:46.220476 26649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:46.220553 26649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:46.221453 26649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "c194c633d5604925a8424888d01259fa"
format_stamp: "Formatted at 2026-09-26 14:55:46 on dist-test-slave-s4p1"
server_key: "de00149db0ca57922b22df374e31699d"
server_key_iv: "b463e4f1f0800ec3da34b56f0f00dc76"
server_key_version: "encryptionkey@0"
I20260926 14:55:46.221741 26649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "c194c633d5604925a8424888d01259fa"
format_stamp: "Formatted at 2026-09-26 14:55:46 on dist-test-slave-s4p1"
server_key: "de00149db0ca57922b22df374e31699d"
server_key_iv: "b463e4f1f0800ec3da34b56f0f00dc76"
server_key_version: "encryptionkey@0"
I20260926 14:55:46.222982 26649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:46.223641 26664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:46.223824 26649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:46.223899 26649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "c194c633d5604925a8424888d01259fa"
format_stamp: "Formatted at 2026-09-26 14:55:46 on dist-test-slave-s4p1"
server_key: "de00149db0ca57922b22df374e31699d"
server_key_iv: "b463e4f1f0800ec3da34b56f0f00dc76"
server_key_version: "encryptionkey@0"
I20260926 14:55:46.223953 26649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:46.227003 26649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:46.228681 26649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:46.228777 26649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:46.232501 26649 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:33501
I20260926 14:55:46.232529 26716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:33501 every 8 connection(s)
I20260926 14:55:46.232841 26649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:46.233731 26717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:46.235771 26717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa: Bootstrap starting.
I20260926 14:55:46.236279 26717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:46.236541 26717 log.cc:824] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:46.237142 26717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa: No bootstrap required, opened a new log
I20260926 14:55:46.238601 26717 raft_consensus.cc:348] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194c633d5604925a8424888d01259fa" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33501 } }
I20260926 14:55:46.238745 26717 raft_consensus.cc:374] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:55:46.238838 26717 raft_consensus.cc:729] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c194c633d5604925a8424888d01259fa, State: Initialized, Role: FOLLOWER
I20260926 14:55:46.238993 26717 consensus_queue.cc:260] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194c633d5604925a8424888d01259fa" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33501 } }
I20260926 14:55:46.239071 26717 raft_consensus.cc:388] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:55:46.239113 26717 raft_consensus.cc:482] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:55:46.239161 26717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:46.239459 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26649
I20260926 14:55:46.239578 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:46.239755 5182 external_mini_cluster.cc:1428] Setting key f42a3eb79ae07db80108f51d641b43b7
I20260926 14:55:46.240092 26717 raft_consensus.cc:504] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194c633d5604925a8424888d01259fa" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33501 } }
I20260926 14:55:46.240275 26717 leader_election.cc:302] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c194c633d5604925a8424888d01259fa; no voters:
I20260926 14:55:46.240477 26717 leader_election.cc:288] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:55:46.240535 26722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:46.240684 26722 raft_consensus.cc:686] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [term 1 LEADER]: Becoming Leader. State: Replica: c194c633d5604925a8424888d01259fa, State: Running, Role: LEADER
I20260926 14:55:46.240692 26717 sys_catalog.cc:564] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:46.240859 26722 consensus_queue.cc:237] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194c633d5604925a8424888d01259fa" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33501 } }
I20260926 14:55:46.241195 26724 sys_catalog.cc:455] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c194c633d5604925a8424888d01259fa. Latest consensus state: current_term: 1 leader_uuid: "c194c633d5604925a8424888d01259fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194c633d5604925a8424888d01259fa" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33501 } } }
I20260926 14:55:46.241345 26724 sys_catalog.cc:458] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:46.241176 26723 sys_catalog.cc:455] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c194c633d5604925a8424888d01259fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194c633d5604925a8424888d01259fa" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 33501 } } }
I20260926 14:55:46.241922 26723 sys_catalog.cc:458] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:46.241923 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:46.240159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45223 (local address 127.5.15.190:33501)
0926 14:55:46.240406 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:46.240415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:46.240448 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:55:46.240557 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:46.240563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:46.240592 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:46.240761 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:46.241318 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:46.241321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:46.241322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:46.241326 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:46.241365 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:46.241378 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:46.241442 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:46.241618 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 14:55:46.241663 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":141,"threads_started":1}
I20260926 14:55:46.242178 26726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:46.242684 26726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:46.242887 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:46.244313 26726 catalog_manager.cc:1347] Generated new cluster ID: 342145778c0d413c9c34f29991a51b11
I20260926 14:55:46.244355 26726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:46.245481 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:46.244337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46423 (local address 127.5.15.190:33501)
0926 14:55:46.244366 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:46.244367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:46.244593 (+ 226us) server_negotiation.cc:407] Connection header received
0926 14:55:46.244663 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:46.244664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:46.244668 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:46.244720 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:46.245292 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:46.245294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:46.245294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:55:46.245297 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:46.245308 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:55:46.245312 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:46.245363 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:46.245414 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:55:46.245427 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 14:55:46.271288 26726 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:46.271487 26726 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:46.281092 26726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c194c633d5604925a8424888d01259fa: Generated new TSK 0
I20260926 14:55:46.281261 26726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:55:46.285516 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26649
2026-09-26T14:55:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1277 ms)
[----------] 18 tests from AuthTokenIssuingTest (23280 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:46 dist-test-slave-s4p1 krb5kdc[26748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:46 dist-test-slave-s4p1 krb5kdc[26748](info): set up 2 sockets
Sep 26 14:55:46 dist-test-slave-s4p1 krb5kdc[26748](info): commencing operation
krb5kdc: starting...
W20260926 14:55:47.306936 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:47.416870 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38073
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38073,127.5.15.189:42937,127.5.15.188:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:47.494515 26761 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:55:47.494730 26761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:47.494774 26761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:47.494829 26761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:47.495899 26761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:47.495959 26761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:47.495982 26761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:47.496012 26761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:47.496037 26761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:47.497445 26761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38073,127.5.15.189:42937,127.5.15.188:40213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:47.497699 26761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:47.497913 26761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:47.502580 26767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:47.502651 26761 server_base.cc:1034] running on GCE node
W20260926 14:55:47.502703 26766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:47.502830 26769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:47.503105 26761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:47.503367 26761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:47.504530 26761 hybrid_clock.cc:648] HybridClock initialized: now 1790434547504522 us; error 32 us; skew 500 ppm
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:47.506894 26761 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:47.507305 26761 webserver.cc:460] Webserver started at http://127.5.15.190:45753/ using document root <none> and password file <none>
I20260926 14:55:47.507485 26761 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:47.507524 26761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:47.507606 26761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:47.508546 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "f28248f61b844323b85cf4d5b52d6c43"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "ded8e54080e93562a0234f195b24db97"
server_key_iv: "f9c214d4373b8239df97797a0bd37a22"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.508831 26761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "f28248f61b844323b85cf4d5b52d6c43"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "ded8e54080e93562a0234f195b24db97"
server_key_iv: "f9c214d4373b8239df97797a0bd37a22"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.509946 26761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:47.510593 26776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:47.510805 26761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:47.510907 26761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "f28248f61b844323b85cf4d5b52d6c43"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "ded8e54080e93562a0234f195b24db97"
server_key_iv: "f9c214d4373b8239df97797a0bd37a22"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.510984 26761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:47.524307 26761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:47.525949 26761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:47.526096 26761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:47.530023 26761 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:38073
I20260926 14:55:47.530056 26828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:38073 every 8 connection(s)
I20260926 14:55:47.530385 26761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:47.531116 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26761
I20260926 14:55:47.531201 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:47.531422 26829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:47.531435 5182 external_mini_cluster.cc:1428] Setting key f4f2cf6aaac31f488a096533710ef1bd
I20260926 14:55:47.532860 26829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.15.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.189@KRBTEST.COM" created.
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:47.545001 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.533558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.5.15.190:38073)
0926 14:55:47.533713 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.533721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.533822 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:55:47.534308 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.534311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.534333 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.534436 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.534941 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.536321 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.537171 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.537692 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.537814 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.542006 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.542013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.542014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.542020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.543549 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.543929 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.543931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.543931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.544003 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.544214 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.544215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.544215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.544401 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.544488 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.544672 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.544819 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260926 14:55:47.545003 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.533527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38073 (local address 127.0.0.1:60106)
0926 14:55:47.533690 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.533726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.534155 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.534451 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.534453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.534689 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.534852 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.534856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.536422 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.536423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.537090 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.537091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.537793 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.537793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.537842 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.538378 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.538386 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.541904 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.543647 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.543653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.543655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.543837 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.544067 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.544068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.544068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.544151 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.544469 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.544474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.544528 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.544625 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.544776 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260926 14:55:47.546022 26829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:47.546356 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.546225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54742)
0926 14:55:47.546274 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.546323 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:55:47.546514 26779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
I20260926 14:55:47.546970 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.546855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54754)
0926 14:55:47.546909 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.546939 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:55:47.547137 26829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:42937: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab.
Entry for principal kudu/127.5.15.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab.
I20260926 14:55:47.554992 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:42937
--webserver_interface=127.5.15.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab
--principal=kudu/127.5.15.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38073,127.5.15.189:42937,127.5.15.188:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:55:47.596354 26829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } attempt: 1
I20260926 14:55:47.596823 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.596695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54758)
0926 14:55:47.596755 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.596795 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:55:47.597143 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.597057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54764)
0926 14:55:47.597100 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.597123 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:55:47.597234 26829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:42937: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
W20260926 14:55:47.634650 26836 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:55:47.634898 26836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:47.634927 26836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:47.634964 26836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:47.636024 26836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:47.636083 26836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:47.636104 26836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:47.636124 26836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:47.636152 26836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:47.637533 26836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38073,127.5.15.189:42937,127.5.15.188:40213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab
--principal=kudu/127.5.15.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:42937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.15.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:47.637794 26836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:47.637995 26836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:47.642516 26841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:47.642524 26842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:47.642704 26836 server_base.cc:1034] running on GCE node
W20260926 14:55:47.642558 26844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:47.643034 26836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:47.643340 26836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:47.644502 26836 hybrid_clock.cc:648] HybridClock initialized: now 1790434547644485 us; error 30 us; skew 500 ppm
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:47.646916 26836 init.cc:377] Logged in from keytab as kudu/127.5.15.189@KRBTEST.COM (short username kudu)
I20260926 14:55:47.647302 26836 webserver.cc:460] Webserver started at http://127.5.15.189:38787/ using document root <none> and password file <none>
I20260926 14:55:47.647526 26836 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:47.647593 26836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:47.647749 26836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:47.648799 26836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data/instance:
uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "39d10c37877c5e54c553a9a7b79a20e4"
server_key_iv: "8c18e0b77f0362980b31ab118e7f6d4e"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.649191 26836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal/instance:
uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "39d10c37877c5e54c553a9a7b79a20e4"
server_key_iv: "8c18e0b77f0362980b31ab118e7f6d4e"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.650441 26836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:47.651152 26851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:47.651293 26836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:47.651405 26836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal
uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "39d10c37877c5e54c553a9a7b79a20e4"
server_key_iv: "8c18e0b77f0362980b31ab118e7f6d4e"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.651494 26836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:47.663807 26836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:47.665422 26836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:47.665385 26829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } attempt: 2
I20260926 14:55:47.665556 26836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:47.665743 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.665623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54780)
0926 14:55:47.665683 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.665716 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:55:47.666121 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.666011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54782)
0926 14:55:47.666077 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.666100 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:55:47.666230 26829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:42937: Network error: Client connection negotiation failed: client connection to 127.5.15.189:42937: connect: Connection refused (error 111)
I20260926 14:55:47.669626 26836 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.189:42937
I20260926 14:55:47.669667 26903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.189:42937 every 8 connection(s)
I20260926 14:55:47.670012 26836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
I20260926 14:55:47.670969 26904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:47.672322 26904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:47.679080 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26836
I20260926 14:55:47.679179 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-1/wal/instance
I20260926 14:55:47.679432 5182 external_mini_cluster.cc:1428] Setting key 13fb261dad56747eef79838d9db00ace
I20260926 14:55:47.680510 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.672914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.5.15.190:38073)
0926 14:55:47.672964 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.672966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.673168 (+ 202us) server_negotiation.cc:407] Connection header received
0926 14:55:47.673585 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.673586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.673597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.673647 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.674138 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.674618 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.675573 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.675753 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.675804 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.677575 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.677576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.677576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.677582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.679154 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.679641 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.679642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.679642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.679692 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.679967 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.679968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.679968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.680054 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.680098 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.680401 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.680460 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:55:47.680794 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.672902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38073 (local address 127.0.0.1:60112)
0926 14:55:47.673055 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.673093 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.673514 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.673692 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.673695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.673910 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.674075 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.674078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.674690 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.674691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.675507 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.675509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.675829 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.675830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.675931 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.676317 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.676324 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.677501 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.679256 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.679263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.679265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.679570 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.679765 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.679766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.679766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.679898 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.680111 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.680116 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.680278 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.680383 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.680542 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260926 14:55:47.681346 26904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.15.188@KRBTEST.COM; defaulting to no policy
I20260926 14:55:47.688700 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.681574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54796)
0926 14:55:47.681618 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.681629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.681668 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.681900 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.681900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.682053 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.682093 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.682094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.683030 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.683031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.683893 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.683896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.684317 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.684318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.684399 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.684720 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.684725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.685928 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.687635 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.687636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.687636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.687790 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.687999 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.687999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.688000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.688050 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.688376 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.688377 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.688413 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.688479 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.688580 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 14:55:47.688773 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.681637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.5.15.189:42937)
0926 14:55:47.681770 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.681774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.681793 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:55:47.681813 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.681815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.681824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.681899 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.682174 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.682905 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.683971 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.684244 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.684296 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.686012 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.686019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.686020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.686026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.687547 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.687880 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.687883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.687883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.687933 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.688131 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.688132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.688132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.688296 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.688396 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.688505 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.688646 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260926 14:55:47.689256 26904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:47.689585 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.689448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55734)
0926 14:55:47.689489 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.689561 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.188@KRBTEST.COM" created.
W20260926 14:55:47.689689 26854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
I20260926 14:55:47.690080 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.689991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55744)
0926 14:55:47.690032 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.690062 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:55:47.690181 26904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:40213: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab.
Entry for principal kudu/127.5.15.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab.
I20260926 14:55:47.703078 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:40213
--webserver_interface=127.5.15.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab
--principal=kudu/127.5.15.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:38073,127.5.15.189:42937,127.5.15.188:40213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:55:47.739415 26904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } attempt: 1
I20260926 14:55:47.739938 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.739764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55760)
0926 14:55:47.739839 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.739902 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:55:47.740239 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.740172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55768)
0926 14:55:47.740199 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.740222 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 14:55:47.740331 26904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:40213: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
I20260926 14:55:47.757356 26829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } attempt: 3
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:55:47.764322 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.757732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.5.15.189:42937)
0926 14:55:47.757795 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.757797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.757808 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:55:47.757860 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.757861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.757879 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.757945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.758327 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.758881 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.759587 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.759814 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.759890 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.761758 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.761760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.761761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.761769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.763193 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.763547 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.763549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.763549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.763605 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.763802 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.763803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.763803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.763921 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.763994 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.764161 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.764234 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:55:47.764323 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.757691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54806)
0926 14:55:47.757730 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.757742 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.757797 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.757951 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.757952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.758132 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.758223 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.758225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.758964 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.758966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.759489 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.759491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.759887 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.759888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.759974 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.760491 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.760496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.761668 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.763279 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.763280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.763280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.763460 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.763674 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.763675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.763675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.763737 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.763998 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.764000 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.764044 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.764124 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.764224 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 14:55:47.764788 26829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:47.765141 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.765037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55782)
0926 14:55:47.765084 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.765112 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:55:47.765542 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.765459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55788)
0926 14:55:47.765503 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.765526 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:55:47.765647 26829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:40213: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
W20260926 14:55:47.782331 26911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:55:47.782508 26911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:47.782526 26911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:47.782580 26911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:47.783701 26911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:47.783741 26911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:47.783761 26911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:47.783808 26911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:47.783829 26911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:47.785243 26911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:38073,127.5.15.189:42937,127.5.15.188:40213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab
--principal=kudu/127.5.15.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.15.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:47.785475 26911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:47.785684 26911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:47.790254 26919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:47.790313 26916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:47.790256 26917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:47.790531 26911 server_base.cc:1034] running on GCE node
I20260926 14:55:47.790737 26911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:47.791054 26911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:47.792186 26911 hybrid_clock.cc:648] HybridClock initialized: now 1790434547792167 us; error 32 us; skew 500 ppm
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:47.794502 26911 init.cc:377] Logged in from keytab as kudu/127.5.15.188@KRBTEST.COM (short username kudu)
I20260926 14:55:47.794931 26911 webserver.cc:460] Webserver started at http://127.5.15.188:38535/ using document root <none> and password file <none>
I20260926 14:55:47.795107 26911 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:47.795147 26911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:47.795228 26911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:47.796146 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data/instance:
uuid: "f6112bab9fa84724a5c6b4bfb94dd86f"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "87e3847a3b080797ea279333792c95f8"
server_key_iv: "6da8000388565f182821e7464a6084de"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.796423 26911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal/instance:
uuid: "f6112bab9fa84724a5c6b4bfb94dd86f"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "87e3847a3b080797ea279333792c95f8"
server_key_iv: "6da8000388565f182821e7464a6084de"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.796795 26829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } attempt: 1
I20260926 14:55:47.797165 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.797051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55798)
0926 14:55:47.797107 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.797137 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:55:47.797500 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.797442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55800)
0926 14:55:47.797466 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.797486 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 14:55:47.797605 26829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:40213: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
I20260926 14:55:47.797986 26911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:47.798688 26926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:47.798866 26911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 14:55:47.798986 26911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal
uuid: "f6112bab9fa84724a5c6b4bfb94dd86f"
format_stamp: "Formatted at 2026-09-26 14:55:47 on dist-test-slave-s4p1"
server_key: "87e3847a3b080797ea279333792c95f8"
server_key_iv: "6da8000388565f182821e7464a6084de"
server_key_version: "encryptionkey@0"
I20260926 14:55:47.799062 26911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:47.808458 26904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } attempt: 2
I20260926 14:55:47.808801 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.808691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55808)
0926 14:55:47.808745 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.808778 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 14:55:47.809100 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.809016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55810)
0926 14:55:47.809056 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.809079 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 14:55:47.809176 26904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:40213: Network error: Client connection negotiation failed: client connection to 127.5.15.188:40213: connect: Connection refused (error 111)
I20260926 14:55:47.815898 26911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:47.817616 26911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:47.817776 26911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:47.821650 26911 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.188:40213
I20260926 14:55:47.821717 26978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.188:40213 every 8 connection(s)
I20260926 14:55:47.822088 26911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
I20260926 14:55:47.823096 26979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:47.824548 26979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:47.827239 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 26911
I20260926 14:55:47.827346 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434201368912-5182-0/minicluster-data/master-2/wal/instance
I20260926 14:55:47.827569 5182 external_mini_cluster.cc:1428] Setting key adc9ae5011222dbdc00db9195306bfd2
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:47.834018 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.825203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.5.15.190:38073)
0926 14:55:47.825264 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.825265 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.825457 (+ 192us) server_negotiation.cc:407] Connection header received
0926 14:55:47.825951 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.825952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.825964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.826016 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.826592 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.827122 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.828488 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.828676 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.828746 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.831000 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.831002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.831002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.831009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.832640 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.833102 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.833103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.833103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.833152 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.833421 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.833422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.833422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.833560 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.833638 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.833895 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.833957 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:55:47.834250 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.825174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:38073 (local address 127.0.0.1:60116)
0926 14:55:47.825351 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.825384 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.825841 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.826089 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.826092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.826315 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.826492 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.826495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.827206 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.827207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.828410 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.828412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.828788 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.828790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.828911 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.829385 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.829398 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.830925 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.832718 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.832725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.832727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.833018 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.833214 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.833215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.833215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.833359 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.833633 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.833638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.833739 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.833858 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.834006 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 14:55:47.834944 26979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:47.836964 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.827952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.5.15.188:40213)
0926 14:55:47.828115 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.828119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.828138 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:55:47.828185 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.828187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.828204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.828287 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.828838 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.829512 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.830472 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.830896 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.830969 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.833805 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.833810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.833810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.833816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.835844 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.836149 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.836152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.836152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.836213 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.836384 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.836384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.836385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.836548 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.836700 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.836788 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.836861 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:55:47.841454 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.835162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:42937 (local address 127.0.0.1:54814)
0926 14:55:47.835205 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.835284 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.835314 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.835443 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.835444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.835558 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.835583 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.835583 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.836433 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.836434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.836881 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.836882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.837380 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.837381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.837437 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.837789 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.837793 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.838871 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.840505 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.840506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.840506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.840688 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.840901 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.840902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.840902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.840957 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.841217 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.841218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.841252 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.841307 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.841368 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 14:55:47.841543 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.835247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.5.15.189:42937)
0926 14:55:47.835296 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.835298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.835308 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:47.835372 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.835373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.835383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.835442 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.835652 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.836367 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.837050 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.837306 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.837383 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.838954 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.838955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.838956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.838963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.840430 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.840774 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.840776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.840776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.840834 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.841039 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.841040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.841040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.841145 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.841223 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.841350 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.841437 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:47.842136 26979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:47.844703 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.837666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.5.15.190:38073)
0926 14:55:47.837730 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.837732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.837853 (+ 121us) server_negotiation.cc:407] Connection header received
0926 14:55:47.837927 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.837928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.837938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.837992 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.838440 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.839009 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.839759 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.839953 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.840031 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.842225 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.842226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.842226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.842233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.843652 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.843961 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.843962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.843962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.844019 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.844185 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.844186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.844186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.844309 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.844498 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.844528 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.844580 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:47.848984 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.842409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.5.15.188:40213)
0926 14:55:47.842641 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.842642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.842648 (+ 6us) server_negotiation.cc:407] Connection header received
0926 14:55:47.842663 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.842663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.842671 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.842714 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.843035 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.843525 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.844406 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.844732 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.844821 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.846406 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.846407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.846407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.846414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.847875 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.848225 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.848226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.848226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.848277 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.848502 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.848502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.848502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.848584 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.848629 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.848794 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.848903 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":217}
I20260926 14:55:47.849032 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.842381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55840)
0926 14:55:47.842436 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.842447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.842482 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.842740 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.842741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.842902 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.842955 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.842957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.843603 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.843605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.844336 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.844338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.844868 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.844869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.844913 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.845290 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.845295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.846322 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.847946 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.847947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.847947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.848146 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.848358 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.848360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.848360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.848435 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.848645 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.848646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.848695 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.848778 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.848865 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:55:47.850597 26979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: Bootstrap starting.
I20260926 14:55:47.851598 26979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:47.851922 26979 log.cc:824] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:47.852782 26979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: No bootstrap required, opened a new log
I20260926 14:55:47.852856 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.845279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.5.15.189:42937)
0926 14:55:47.845321 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.845323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.845332 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:47.845386 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.845387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.845395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.845442 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.845880 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.846396 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.847060 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.847369 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.847439 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.849981 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.849982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.849983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.849989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.851664 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.852067 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.852068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.852069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.852126 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.852352 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.852353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.852353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.852498 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.852649 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.852689 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.852766 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:55:47.854360 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:47.854722 26979 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.854902 26979 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6112bab9fa84724a5c6b4bfb94dd86f, State: Initialized, Role: FOLLOWER
I20260926 14:55:47.855005 26979 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.855338 26979 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:47.855316 26985 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.855400 26985 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:47.855458 26803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f28248f61b844323b85cf4d5b52d6c43"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60120
W20260926 14:55:47.857190 26996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:47.857242 26996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:47.857390 26911 master_runner.cc:186] Error getting master registration for 127.5.15.190:38073: OK, instance_id { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" instance_seqno: 1790434547528887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:55:47.860220 26911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:55:47.872778 26829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } attempt: 2
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:47.880316 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.873222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.5.15.188:40213)
0926 14:55:47.873277 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.873279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.873288 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:47.873311 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.873313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.873322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.873371 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.873687 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.874401 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.875333 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.875548 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.875629 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.877488 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.877489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.877489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.877497 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.879076 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.879492 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.879494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.879494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.879558 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.879762 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.879763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.879763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.879916 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.879996 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.880147 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.880221 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:55:47.880316 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.873119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55850)
0926 14:55:47.873170 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.873182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.873229 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.873375 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.873375 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.873542 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.873585 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.873587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.874471 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.874472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.875251 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.875254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.875626 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.875627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.875683 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.876173 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.876178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.877391 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.879176 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.879177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.879178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.879414 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.879634 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.879635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.879635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.879698 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.879987 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.879988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.880035 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.880123 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.880221 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 14:55:47.882215 26829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43: Bootstrap starting.
I20260926 14:55:47.882989 26829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:47.883297 26829 log.cc:824] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:47.884155 26829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43: No bootstrap required, opened a new log
I20260926 14:55:47.885483 26829 raft_consensus.cc:348] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.885617 26829 raft_consensus.cc:729] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28248f61b844323b85cf4d5b52d6c43, State: Initialized, Role: FOLLOWER
I20260926 14:55:47.885701 26829 consensus_queue.cc:260] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.885978 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.886024 26829 sys_catalog.cc:564] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:47.886132 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:47.886040 26803 raft_consensus.cc:482] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:55:47.886831 26803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:47.887604 26803 raft_consensus.cc:504] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.888269 26803 leader_election.cc:288] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [CANDIDATE]: Term 1 election: Requested vote from peers 8ffb0f8255ab46c6b3c67f93a72b0cd2 (127.5.15.189:42937), f6112bab9fa84724a5c6b4bfb94dd86f (127.5.15.188:40213)
W20260926 14:55:47.888324 27010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:47.888486 27010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:47.888592 26953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f28248f61b844323b85cf4d5b52d6c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f"
I20260926 14:55:47.888723 26953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:47.889565 26953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f28248f61b844323b85cf4d5b52d6c43 in term 1.
I20260926 14:55:47.889791 26779 leader_election.cc:302] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f28248f61b844323b85cf4d5b52d6c43, f6112bab9fa84724a5c6b4bfb94dd86f; no voters:
I20260926 14:55:47.889899 26999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:47.890084 26999 raft_consensus.cc:686] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [term 1 LEADER]: Becoming Leader. State: Replica: f28248f61b844323b85cf4d5b52d6c43, State: Running, Role: LEADER
I20260926 14:55:47.890169 26999 consensus_queue.cc:237] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.890558 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28248f61b844323b85cf4d5b52d6c43. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.890640 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:47.890851 26761 master_runner.cc:186] Error getting master registration for 127.5.15.189:42937: OK, instance_id { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" instance_seqno: 1790434547668540 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:55:47.893359 26761 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:55:47.899042 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.891324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.5.15.188:40213)
0926 14:55:47.891357 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.891358 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.891569 (+ 211us) server_negotiation.cc:407] Connection header received
0926 14:55:47.891659 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.891660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.891669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.891697 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.892428 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.893156 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.894202 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.894489 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.894527 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.896063 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.896064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.896065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.896071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.897690 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.898150 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.898151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.898152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.898224 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.898421 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.898422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.898422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.898538 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.898620 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.898869 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.898946 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 14:55:47.899323 27018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:47.899523 27018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:47.900292 26904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } attempt: 3
I20260926 14:55:47.901448 26953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [term 1 FOLLOWER]: Refusing update from remote peer f28248f61b844323b85cf4d5b52d6c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:47.901782 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.891006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.5.15.189:42937)
0926 14:55:47.891047 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.891049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.891358 (+ 309us) server_negotiation.cc:407] Connection header received
0926 14:55:47.891503 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.891504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.891526 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.891556 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.892738 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.893424 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.894947 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.895707 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.895753 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.898064 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.898065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.898065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.898073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.900405 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.900889 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.900889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.900890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.900950 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.901158 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.901159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.901159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.901279 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.901343 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.901622 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.901695 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:47.901794 26999 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:47.903149 26985 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28248f61b844323b85cf4d5b52d6c43. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.903234 26985 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 14:55:47 dist-test-slave-s4p1 krb5kdc[26748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:47.905367 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.905444 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:47.905783 26985 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.905857 26985 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:47.906221 27021 mvcc.cc:205] Tried to move back new op lower bound from 7333619908201361408 to 7333619908158775296. Current Snapshot: MvccSnapshot[applied={T|T < 7333619908201361408}]
I20260926 14:55:47.906798 27022 mvcc.cc:205] Tried to move back new op lower bound from 7333619908201361408 to 7333619908158775296. Current Snapshot: MvccSnapshot[applied={T|T < 7333619908201361408}]
I20260926 14:55:47.907593 27018 catalog_manager.cc:1347] Generated new cluster ID: ec585df18e714f8a9e1fb4aa0c6992ca
I20260926 14:55:47.907653 27018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:47.908640 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.900552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:40213 (local address 127.0.0.1:55870)
0926 14:55:47.900638 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:47.900648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:47.900684 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:47.900938 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:47.900939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:47.901091 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:47.901135 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.901137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.902053 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.902053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.902637 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:47.902640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.903176 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.903177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:47.903232 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.903739 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:47.903744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:47.905235 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:47.907551 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.907552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.907552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.907769 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.908017 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:47.908018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:47.908018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:47.908069 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:47.908302 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:47.908304 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:47.908371 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 14:55:47.908455 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:55:47.908544 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 14:55:47.908929 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.900768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.5.15.188:40213)
0926 14:55:47.900803 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.900806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.900812 (+ 6us) server_negotiation.cc:407] Connection header received
0926 14:55:47.900830 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.900831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.900841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.900888 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.901210 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.901897 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.902729 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.903031 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.903109 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.905332 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.905333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.905333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.905340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.907387 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.907898 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.907899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.907899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.907955 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.908121 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.908122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.908122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.908222 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.908279 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.908502 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.908586 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:47.910163 26904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: Bootstrap starting.
I20260926 14:55:47.910837 26904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:47.911113 26904 log.cc:824] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:47.915922 26904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: No bootstrap required, opened a new log
I20260926 14:55:47.917732 26904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.917862 26904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffb0f8255ab46c6b3c67f93a72b0cd2, State: Initialized, Role: FOLLOWER
I20260926 14:55:47.917932 26904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } }
I20260926 14:55:47.918228 26904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:47.918452 26877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:47.919392 26877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [term 1 FOLLOWER]: Refusing update from remote peer f28248f61b844323b85cf4d5b52d6c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:47.919691 27012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:47.919734 26878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f28248f61b844323b85cf4d5b52d6c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2"
I20260926 14:55:47.920979 26878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f28248f61b844323b85cf4d5b52d6c43 in term 1.
I20260926 14:55:47.925354 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:47.907071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.5.15.190:38073)
0926 14:55:47.907112 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.907113 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.907266 (+ 153us) server_negotiation.cc:407] Connection header received
0926 14:55:47.907360 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.907361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.907372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.907403 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.915395 (+ 7992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.916165 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.917944 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:47.918228 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:47.918272 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:47.922485 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:47.922487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:47.922487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:47.922494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:47.924183 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.924561 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.924562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.924562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.924611 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:47.924814 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:47.924815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:47.924815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:47.924932 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:47.924998 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:47.925191 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 14:55:47.925265 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:55:47.926909 27027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.927003 27027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:47.927053 27027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f28248f61b844323b85cf4d5b52d6c43. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.927073 27039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.927106 27027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:47.927141 27039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:47.927516 27040 mvcc.cc:205] Tried to move back new op lower bound from 7333619908201361408 to 7333619908158775296. Current Snapshot: MvccSnapshot[applied={T|T < 7333619908201361408}]
I20260926 14:55:47.927898 27018 catalog_manager.cc:1370] Generated new certificate authority record
W20260926 14:55:47.927985 27038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:47.928037 27038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:47.928301 26999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f28248f61b844323b85cf4d5b52d6c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f28248f61b844323b85cf4d5b52d6c43" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 38073 } } peers { permanent_uuid: "8ffb0f8255ab46c6b3c67f93a72b0cd2" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 42937 } } peers { permanent_uuid: "f6112bab9fa84724a5c6b4bfb94dd86f" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 40213 } } }
I20260926 14:55:47.928409 26999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:47.928817 27018 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:47.938994 27018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f28248f61b844323b85cf4d5b52d6c43: Generated new TSK 0
I20260926 14:55:47.939198 27018 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:55:47.963551 26833 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.962035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.5.15.190:38073)
0926 14:55:47.962087 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.962089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.962938 (+ 849us) server_negotiation.cc:407] Connection header received
0926 14:55:47.963038 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.963039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.963049 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.963079 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.963488 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60150: BlockingRecv error: recv got EOF from 127.0.0.1:60150 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 14:55:47.964234 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.962401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.5.15.189:42937)
0926 14:55:47.962437 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.962439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.963769 (+ 1330us) server_negotiation.cc:407] Connection header received
0926 14:55:47.963857 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.963858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.963868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.963895 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.964177 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54834: BlockingRecv error: recv got EOF from 127.0.0.1:54834 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 14:55:47.964924 26983 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.962666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.5.15.188:40213)
0926 14:55:47.962702 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.962703 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.964434 (+ 1731us) server_negotiation.cc:407] Connection header received
0926 14:55:47.964552 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.964553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.964562 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.964590 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.964858 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55872: BlockingRecv error: recv got EOF from 127.0.0.1:55872 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 14:55:47.967839 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.967146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.5.15.189:42937)
0926 14:55:47.967176 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.967178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.967346 (+ 168us) server_negotiation.cc:407] Connection header received
0926 14:55:47.967440 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.967441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.967457 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.967484 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.967797 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54840: BlockingRecv error: recv got EOF from 127.0.0.1:54840 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260926 14:55:47.970729 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.970011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.5.15.189:42937)
0926 14:55:47.970050 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.970052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.970279 (+ 227us) server_negotiation.cc:407] Connection header received
0926 14:55:47.970368 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.970368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.970377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.970404 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.970692 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54846: BlockingRecv error: recv got EOF from 127.0.0.1:54846 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 14:55:47.974373 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.973694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.5.15.189:42937)
0926 14:55:47.973723 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.973724 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.973946 (+ 222us) server_negotiation.cc:407] Connection header received
0926 14:55:47.974035 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.974036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.974044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.974070 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.974333 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54852: BlockingRecv error: recv got EOF from 127.0.0.1:54852 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 14:55:47.980033 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.979332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.5.15.189:42937)
0926 14:55:47.979362 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.979364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.979588 (+ 224us) server_negotiation.cc:407] Connection header received
0926 14:55:47.979674 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.979675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.979684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.979710 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.979997 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54854: BlockingRecv error: recv got EOF from 127.0.0.1:54854 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 14:55:47.989694 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:47.988984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.5.15.189:42937)
0926 14:55:47.989015 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:47.989017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:47.989249 (+ 232us) server_negotiation.cc:407] Connection header received
0926 14:55:47.989335 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:47.989336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:47.989345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:47.989371 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:47.989645 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54868: BlockingRecv error: recv got EOF from 127.0.0.1:54868 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 14:55:48.007802 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:48.006840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.5.15.189:42937)
0926 14:55:48.006878 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:48.006879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:48.007168 (+ 289us) server_negotiation.cc:407] Connection header received
0926 14:55:48.007292 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:48.007294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:48.007305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:48.007335 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:48.007747 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54870: BlockingRecv error: recv got EOF from 127.0.0.1:54870 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 14:55:48.042006 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:48.041211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.5.15.189:42937)
0926 14:55:48.041268 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:48.041271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:48.041385 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:55:48.041453 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:48.041454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:48.041479 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:48.041541 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:48.041947 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54880: BlockingRecv error: recv got EOF from 127.0.0.1:54880 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 14:55:48.107769 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:48.107108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.5.15.189:42937)
0926 14:55:48.107163 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:48.107165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:48.107261 (+ 96us) server_negotiation.cc:407] Connection header received
0926 14:55:48.107349 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:48.107350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:48.107361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:48.107407 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:48.107721 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54890: BlockingRecv error: recv got EOF from 127.0.0.1:54890 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 14:55:48.237562 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:48.236927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.5.15.189:42937)
0926 14:55:48.236990 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:48.236993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:48.237032 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:55:48.237059 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:48.237060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:48.237070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:48.237116 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:48.237472 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54906: BlockingRecv error: recv got EOF from 127.0.0.1:54906 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 14:55:48.495380 26908 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:48.494813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.5.15.189:42937)
0926 14:55:48.494881 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:48.494883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:48.494893 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:48.494941 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:48.494942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:48.494951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:48.495003 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:48.495304 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54914: BlockingRecv error: recv got EOF from 127.0.0.1:54914 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:55:48.857661 26996 catalog_manager.cc:1259] Loaded cluster ID: ec585df18e714f8a9e1fb4aa0c6992ca
I20260926 14:55:48.857735 26996 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: loading cluster ID for follower catalog manager: success
I20260926 14:55:48.858598 26996 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: acquiring CA information for follower catalog manager: success
I20260926 14:55:48.858945 26996 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6112bab9fa84724a5c6b4bfb94dd86f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:55:48.928480 27038 catalog_manager.cc:1259] Loaded cluster ID: ec585df18e714f8a9e1fb4aa0c6992ca
I20260926 14:55:48.928555 27038 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: loading cluster ID for follower catalog manager: success
I20260926 14:55:48.929323 27038 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: acquiring CA information for follower catalog manager: success
I20260926 14:55:48.929584 27038 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ffb0f8255ab46c6b3c67f93a72b0cd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:55:49.010982 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:49.008548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.5.15.189:42937)
0926 14:55:49.008660 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:49.008661 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:49.008670 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:49.008718 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:49.008719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:49.008766 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:49.008838 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:49.008977 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.009457 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.010200 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.010383 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.010440 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:49.010568 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:49.010676 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:49.010734 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:49.010850 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 14:55:49.010868 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260926 14:55:49.015024 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:49.012802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.5.15.188:40213)
0926 14:55:49.012901 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:49.012903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:49.012911 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:55:49.012944 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:49.012945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:49.012976 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:49.013076 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:49.013173 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.013648 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.014366 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.014544 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.014593 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:49.014711 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:49.014853 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:49.014908 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:49.014950 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:55:49.014966 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":52,"threads_started":1}
I20260926 14:55:49.019696 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:49.016580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.5.15.189:42937)
0926 14:55:49.016634 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:49.016636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:49.016870 (+ 234us) server_negotiation.cc:407] Connection header received
0926 14:55:49.016952 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:49.016953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:49.016962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:49.017007 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:49.017182 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.017924 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.018863 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.019112 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.019191 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:49.019386 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:49.019432 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:49.019460 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:49.019579 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 14:55:49.019606 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 14:55:49.019728 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:49.016852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.5.15.188:40213)
0926 14:55:49.016890 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:49.016892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:49.016903 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:55:49.017001 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:49.017002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:49.017012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:49.017063 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:49.017207 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.017901 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.018957 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.019243 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.019303 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:49.019466 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:49.019500 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:49.019530 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:49.019638 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 14:55:49.019659 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 14:55:49.020238 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:49.016539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.5.15.190:38073)
0926 14:55:49.016633 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:49.016635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:49.016716 (+ 81us) server_negotiation.cc:407] Connection header received
0926 14:55:49.016790 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:49.016791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:49.016843 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:49.016941 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:49.017107 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.017679 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.018820 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:49.019145 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:49.019204 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:49.019430 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:49.019550 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:49.019613 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:49.020100 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 14:55:49.020134 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260926 14:55:49.021999 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26761
I20260926 14:55:49.027146 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26836
I20260926 14:55:49.031981 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 26911
2026-09-26T14:55:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2748 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:55:49 dist-test-slave-s4p1 krb5kdc[27128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:55:49 dist-test-slave-s4p1 krb5kdc[27128](info): set up 2 sockets
Sep 26 14:55:49 dist-test-slave-s4p1 krb5kdc[27128](info): commencing operation
krb5kdc: starting...
W20260926 14:55:50.055143 5182 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/test-user.keytab.
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.15.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
Entry for principal kudu/127.5.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab.
I20260926 14:55:50.165594 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36421
--webserver_interface=127.5.15.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36421,127.5.15.189:39063,127.5.15.188:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:55:50.243736 27141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:55:50.243964 27141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:50.243993 27141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:50.244036 27141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:50.245087 27141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:50.245147 27141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:50.245169 27141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:50.245189 27141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:50.245218 27141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:50.246618 27141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36421,127.5.15.189:39063,127.5.15.188:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.190.keytab
--principal=kudu/127.5.15.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.190:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.15.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:50.246894 27141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:50.247105 27141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:50.251565 27147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:50.251590 27146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:50.251595 27149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:50.251861 27141 server_base.cc:1034] running on GCE node
I20260926 14:55:50.252092 27141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:50.252408 27141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:50.253569 27141 hybrid_clock.cc:648] HybridClock initialized: now 1790434550253554 us; error 44 us; skew 500 ppm
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:50.256037 27141 init.cc:377] Logged in from keytab as kudu/127.5.15.190@KRBTEST.COM (short username kudu)
I20260926 14:55:50.256470 27141 webserver.cc:460] Webserver started at http://127.5.15.190:42575/ using document root <none> and password file <none>
I20260926 14:55:50.256698 27141 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:50.256781 27141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:50.256944 27141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:50.257929 27141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data/instance:
uuid: "67d58db2daa744efbc53d115cc07f700"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "d2785fba81f8dc904a6fdf6a29467c56"
server_key_iv: "51490ef3f7c8ba05a84f81e92f6c47da"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.258288 27141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal/instance:
uuid: "67d58db2daa744efbc53d115cc07f700"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "d2785fba81f8dc904a6fdf6a29467c56"
server_key_iv: "51490ef3f7c8ba05a84f81e92f6c47da"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.259953 27141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 14:55:50.260682 27156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:50.260874 27141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:50.260968 27141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal
uuid: "67d58db2daa744efbc53d115cc07f700"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "d2785fba81f8dc904a6fdf6a29467c56"
server_key_iv: "51490ef3f7c8ba05a84f81e92f6c47da"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.261050 27141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:50.270112 27141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:50.272351 27141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:50.272500 27141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:50.276275 27141 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.190:36421
I20260926 14:55:50.276301 27208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.190:36421 every 8 connection(s)
I20260926 14:55:50.276605 27141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/data/info.pb
I20260926 14:55:50.277518 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:50.278982 27209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:50.279379 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 27141
I20260926 14:55:50.279490 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-0/wal/instance
I20260926 14:55:50.279700 5182 external_mini_cluster.cc:1428] Setting key f8527590abd2f6ba6045f540036c567c
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.15.189@KRBTEST.COM; defaulting to no policy
I20260926 14:55:50.288736 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.279720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.5.15.190:36421)
0926 14:55:50.279894 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.279903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.279978 (+ 75us) server_negotiation.cc:407] Connection header received
0926 14:55:50.280479 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.280482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.280504 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.280607 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.281199 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.282048 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.283000 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.283563 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.283669 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.285886 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.285892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.285892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.285898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.287438 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.287774 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.287776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.287776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.287859 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.288071 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.288072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.288072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.288255 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.288318 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.288459 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.288539 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260926 14:55:50.288746 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.279698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36421 (local address 127.0.0.1:55190)
0926 14:55:50.279864 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.279902 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.280335 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.280621 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.280623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.280855 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.281089 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.281095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.282136 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.282136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.282906 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.282908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.283669 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.283670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.283722 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.284208 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.284216 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.285807 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.287509 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.287515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.287516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.287695 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.287934 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.287936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.287936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.288010 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.288314 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.288319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.288369 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.288435 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.288638 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":67,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.189@KRBTEST.COM" created.
I20260926 14:55:50.289659 27209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:50.289980 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.289871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:39550)
0926 14:55:50.289919 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.289961 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:55:50.290127 27158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
I20260926 14:55:50.290599 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.290492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:39558)
0926 14:55:50.290542 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.290571 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:55:50.290838 27209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:39063: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab.
Entry for principal kudu/127.5.15.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab.
I20260926 14:55:50.303403 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:39063
--webserver_interface=127.5.15.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab
--principal=kudu/127.5.15.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36421,127.5.15.189:39063,127.5.15.188:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:55:50.340051 27209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } attempt: 1
I20260926 14:55:50.340467 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.340350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:39568)
0926 14:55:50.340405 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.340440 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:55:50.340826 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.340737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:39572)
0926 14:55:50.340782 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.340804 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:55:50.340909 27209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:39063: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
W20260926 14:55:50.385532 27216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:55:50.385710 27216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:50.385727 27216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:50.385773 27216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:50.386932 27216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:50.386972 27216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:50.386986 27216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:50.386999 27216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:50.387045 27216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:50.388468 27216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36421,127.5.15.189:39063,127.5.15.188:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.189.keytab
--principal=kudu/127.5.15.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.189:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.15.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:50.388674 27216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:50.388863 27216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:55:50.393649 27221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:50.393709 27222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:50.393911 27224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:50.394057 27216 server_base.cc:1034] running on GCE node
I20260926 14:55:50.394315 27216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:50.394614 27216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:50.395780 27216 hybrid_clock.cc:648] HybridClock initialized: now 1790434550395770 us; error 41 us; skew 500 ppm
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:50.398094 27216 init.cc:377] Logged in from keytab as kudu/127.5.15.189@KRBTEST.COM (short username kudu)
I20260926 14:55:50.398536 27216 webserver.cc:460] Webserver started at http://127.5.15.189:34945/ using document root <none> and password file <none>
I20260926 14:55:50.398778 27216 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:50.398854 27216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:50.399001 27216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:50.399986 27216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data/instance:
uuid: "2cfa0f16261e4167a5c9be7e916b13bd"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "c88f7327d86976f10c4d884420546623"
server_key_iv: "9bdc74d27daaddf3ecf06871c781a7de"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.400310 27216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal/instance:
uuid: "2cfa0f16261e4167a5c9be7e916b13bd"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "c88f7327d86976f10c4d884420546623"
server_key_iv: "9bdc74d27daaddf3ecf06871c781a7de"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.401543 27216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 14:55:50.402208 27231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:50.402387 27216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:50.402478 27216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal
uuid: "2cfa0f16261e4167a5c9be7e916b13bd"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "c88f7327d86976f10c4d884420546623"
server_key_iv: "9bdc74d27daaddf3ecf06871c781a7de"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.402557 27216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:50.409032 27209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } attempt: 2
I20260926 14:55:50.409382 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.409266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:39588)
0926 14:55:50.409322 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.409356 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:55:50.409667 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.409583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:39590)
0926 14:55:50.409624 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.409646 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 14:55:50.409754 27209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.189:39063: Network error: Client connection negotiation failed: client connection to 127.5.15.189:39063: connect: Connection refused (error 111)
I20260926 14:55:50.415491 27216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:50.417587 27216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:50.417730 27216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:50.421446 27216 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.189:39063
I20260926 14:55:50.421490 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.189:39063 every 8 connection(s)
I20260926 14:55:50.421801 27216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/data/info.pb
I20260926 14:55:50.422708 27284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:50.424114 27284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:50.427415 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 27216
I20260926 14:55:50.427524 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-1/wal/instance
I20260926 14:55:50.427733 5182 external_mini_cluster.cc:1428] Setting key e2a5590df2435cdb2667a26e0a7e4c09
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:50.432711 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.424691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.5.15.190:36421)
0926 14:55:50.424751 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.424754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.424952 (+ 198us) server_negotiation.cc:407] Connection header received
0926 14:55:50.425361 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.425362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.425371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.425420 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.425957 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.426435 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.427540 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.427755 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.427843 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.429794 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.429795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.429796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.429804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.431283 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.431776 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.431777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.431777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.431826 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.432105 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.432106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.432106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.432203 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.432250 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.432588 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.432653 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:55:50.433023 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.424673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36421 (local address 127.0.0.1:60952)
0926 14:55:50.424847 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.424879 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.425287 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.425489 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.425492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.425708 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.425890 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.425893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.426512 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.426512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.427455 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.427458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.427871 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.427872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.427990 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.428445 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.428451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.429706 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.431364 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.431371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.431374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.431695 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.431890 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.431891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.431891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.432035 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.432268 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.432275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.432447 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.432572 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.432747 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260926 14:55:50.433622 27284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.15.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.15.188@KRBTEST.COM" created.
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:55:50.441419 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.433932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.5.15.189:39063)
0926 14:55:50.434091 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.434094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.434109 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:55:50.434126 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.434127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.434137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.434217 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.434477 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.435240 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.435937 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.436223 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.436301 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.438561 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.438570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.438570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.438578 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.440218 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.440587 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.440589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.440589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.440638 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.440855 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.440855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.440855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.440976 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.441036 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.441194 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.441257 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260926 14:55:50.441519 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.433863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:48232)
0926 14:55:50.433921 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.433930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.433971 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.434217 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.434218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.434355 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.434396 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.434397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.435326 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.435327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.435839 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.435840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.436322 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.436322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.436385 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.436721 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.436725 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.438483 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.440290 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.440291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.440292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.440508 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.440719 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.440720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.440720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.440795 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.441043 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.441043 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.441090 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.441173 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.441327 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":51}
I20260926 14:55:50.442093 27284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:50.442443 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.442309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49506)
0926 14:55:50.442361 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.442414 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:55:50.442641 27234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
I20260926 14:55:50.443082 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.442953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49510)
0926 14:55:50.443010 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.443051 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:55:50.443239 27284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:43111: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.15.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab.
Entry for principal kudu/127.5.15.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab.
I20260926 14:55:50.452803 5182 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmCUWsf/build/release/bin/kudu
/tmp/dist-test-taskmCUWsf/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:43111
--webserver_interface=127.5.15.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab
--principal=kudu/127.5.15.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.15.148:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.15.190:36421,127.5.15.189:39063,127.5.15.188:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:55:50.492476 27284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } attempt: 1
I20260926 14:55:50.492995 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.492857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49512)
0926 14:55:50.492917 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.492962 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:55:50.493359 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.493269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49518)
0926 14:55:50.493314 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.493341 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:55:50.493458 27284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:43111: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
I20260926 14:55:50.500982 27209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } attempt: 3
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:55:50.508107 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.501352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.5.15.189:39063)
0926 14:55:50.501420 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.501422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.501431 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:50.501457 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.501458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.501472 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.501538 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.501843 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.502324 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.503125 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.503341 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.503429 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.505465 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.505466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.505466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.505472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.507091 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.507470 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.507471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.507471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.507518 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.507705 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.507706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.507706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.507796 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.507842 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.507993 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.508052 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:55:50.508150 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.501258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:48244)
0926 14:55:50.501316 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.501333 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.501389 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.501542 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.501543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.501699 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.501765 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.501767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.502450 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.502452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.503040 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.503042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.503417 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.503418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.503497 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.503963 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.503967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.505389 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.507165 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.507166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.507167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.507395 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.507581 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.507581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.507582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.507647 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.507855 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.507855 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.507902 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.507970 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.508060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:55:50.508549 27209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:50.508882 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.508773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49526)
0926 14:55:50.508820 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.508853 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 14:55:50.509202 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.509116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49530)
0926 14:55:50.509159 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.509184 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 14:55:50.509290 27209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:43111: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
W20260926 14:55:50.533944 27291 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:55:50.534128 27291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:55:50.534144 27291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:55:50.534162 27291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:55:50.535239 27291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:55:50.535281 27291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:55:50.535296 27291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:55:50.535308 27291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:55:50.535355 27291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:55:50.536793 27291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.15.148:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.15.190:36421,127.5.15.189:39063,127.5.15.188:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/krb5kdc/kudu_127.5.15.188.keytab
--principal=kudu/127.5.15.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.15.188:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.15.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 14:55:50.536957 27291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:55:50.537176 27291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:55:50.540442 27209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } attempt: 1
I20260926 14:55:50.540835 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.540717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49544)
0926 14:55:50.540772 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.540801 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:55:50.541251 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.541169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49554)
0926 14:55:50.541198 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.541222 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 14:55:50.541368 27209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:43111: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
W20260926 14:55:50.541630 27299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:55:50.541647 27296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:50.541918 27291 server_base.cc:1034] running on GCE node
W20260926 14:55:50.541635 27297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:55:50.542183 27291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:55:50.542477 27291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:55:50.543634 27291 hybrid_clock.cc:648] HybridClock initialized: now 1790434550543615 us; error 33 us; skew 500 ppm
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:55:50.545877 27291 init.cc:377] Logged in from keytab as kudu/127.5.15.188@KRBTEST.COM (short username kudu)
I20260926 14:55:50.546307 27291 webserver.cc:460] Webserver started at http://127.5.15.188:44273/ using document root <none> and password file <none>
I20260926 14:55:50.546479 27291 fs_manager.cc:359] Metadata directory not provided
I20260926 14:55:50.546517 27291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:55:50.546600 27291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:55:50.547590 27291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data/instance:
uuid: "f177d0f2db504e77b80dfdef00d9b343"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "588496a18b0bbb18288bf4ed42877f8a"
server_key_iv: "a1c696e184f8ed7a357587d446fbfa54"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.547869 27291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal/instance:
uuid: "f177d0f2db504e77b80dfdef00d9b343"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "588496a18b0bbb18288bf4ed42877f8a"
server_key_iv: "a1c696e184f8ed7a357587d446fbfa54"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.549018 27291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:55:50.549654 27306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:55:50.549834 27291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 14:55:50.549921 27291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data,/tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal
uuid: "f177d0f2db504e77b80dfdef00d9b343"
format_stamp: "Formatted at 2026-09-26 14:55:50 on dist-test-slave-s4p1"
server_key: "588496a18b0bbb18288bf4ed42877f8a"
server_key_iv: "a1c696e184f8ed7a357587d446fbfa54"
server_key_version: "encryptionkey@0"
I20260926 14:55:50.549993 27291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:55:50.559388 27291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:55:50.561080 27291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:55:50.561218 27291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:55:50.561616 27284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } attempt: 2
I20260926 14:55:50.562070 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.561910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49560)
0926 14:55:50.561975 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.562021 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 14:55:50.562466 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.562376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49568)
0926 14:55:50.562408 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.562438 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 14:55:50.562621 27284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.15.188:43111: Network error: Client connection negotiation failed: client connection to 127.5.15.188:43111: connect: Connection refused (error 111)
I20260926 14:55:50.565649 27291 rpc_server.cc:307] RPC server started. Bound to: 127.5.15.188:43111
I20260926 14:55:50.565678 27358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.15.188:43111 every 8 connection(s)
I20260926 14:55:50.566004 27291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/data/info.pb
I20260926 14:55:50.567009 27359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:55:50.567441 5182 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmCUWsf/build/release/bin/kudu as pid 27291
I20260926 14:55:50.567552 5182 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmCUWsf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434201368912-5182-0/minicluster-data/master-2/wal/instance
I20260926 14:55:50.567755 5182 external_mini_cluster.cc:1428] Setting key 72aebc8ba121913202a1dec768ad55a0
I20260926 14:55:50.568825 27359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:50.577880 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.569529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.5.15.190:36421)
0926 14:55:50.569575 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.569577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.569689 (+ 112us) server_negotiation.cc:407] Connection header received
0926 14:55:50.569791 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.569792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.569802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.569865 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.570307 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.571032 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.572260 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.572537 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.572603 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.574907 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.574909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.574909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.574917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.576217 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.576661 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.576662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.576663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.576721 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.576961 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.576962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.576962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.577073 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.577152 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.577681 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.577765 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 14:55:50.577961 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.569421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.190:36421 (local address 127.0.0.1:60960)
0926 14:55:50.569587 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.569631 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.569707 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.569879 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.569881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.570125 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.570212 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.570216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.571145 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.571146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.572190 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.572192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.572645 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.572646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.572709 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.573214 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.573225 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.574825 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.576324 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.576331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.576333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.576586 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.576789 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.576790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.576791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.576893 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.577384 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.577390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.577530 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.577640 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.577769 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260926 14:55:50.578246 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.568259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.5.15.188:43111)
0926 14:55:50.568423 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.568431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.568454 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:55:50.568827 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.568833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.568875 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.568985 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.569808 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.571074 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.572121 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.572529 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.572633 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.575460 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.575466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.575466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.575472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.577066 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.577382 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.577384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.577384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.577435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.577651 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.577651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.577652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.577809 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.577962 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.578060 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.578138 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 14:55:50.579015 27359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.190@KRBTEST.COM
I20260926 14:55:50.585780 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.579298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.189:39063 (local address 127.0.0.1:48258)
0926 14:55:50.579362 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.579373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.579416 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.579583 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.579583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.579686 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.579728 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.579728 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.580348 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.580349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.580833 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.580835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.581217 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.581218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.581280 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.581798 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.581802 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.583065 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.584780 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.584781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.584781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.584948 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.585161 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.585162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.585163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.585217 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.585479 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.585480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.585520 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.585607 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.585695 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:55:50.585789 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.579386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.5.15.189:39063)
0926 14:55:50.579464 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.579466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.579474 (+ 8us) server_negotiation.cc:407] Connection header received
0926 14:55:50.579499 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.579500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.579511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.579582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.579793 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.580270 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.580918 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.581146 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.581221 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.583145 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.583146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.583146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.583152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.584695 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.585033 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.585034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.585034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.585088 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.585303 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.585304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.585304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.585415 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.585486 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.585632 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.585698 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:55:50.586217 27359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:55:50.586290 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.579176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.5.15.190:36421)
0926 14:55:50.579232 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.579234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.579241 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:55:50.579261 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.579261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.579271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.579324 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.579886 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.580286 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.580966 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.581156 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.581227 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.583752 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.583753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.583753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.583760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.585166 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.585514 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.585515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.585515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.585572 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.585766 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.585767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.585767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.585893 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.586070 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.586132 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.586206 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.188@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.15.189@KRBTEST.COM
I20260926 14:55:50.592856 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.586422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49590)
0926 14:55:50.586487 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.586495 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.586526 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.586628 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.586629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.586797 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.586828 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.586830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.587420 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.587421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.587876 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.587877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.588262 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.588263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.588310 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.588703 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.588708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.590080 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.591847 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.591848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.591848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.592098 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.592284 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.592285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.592286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.592352 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.592578 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.592579 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.592624 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.592706 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.592796 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 14:55:50.592978 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.586499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.5.15.188:43111)
0926 14:55:50.586534 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.586536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.586546 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:50.586565 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.586566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.586574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.586623 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.586897 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.587330 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.587975 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.588182 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.588262 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.590168 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.590169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.590169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.590174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.591774 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.592172 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.592173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.592173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.592220 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.592426 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.592427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.592427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.592515 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.592561 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.592768 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.592858 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":20}
I20260926 14:55:50.593950 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.586823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.5.15.189:39063)
0926 14:55:50.586854 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.586856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.586901 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:55:50.586968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.586969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.586978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.587031 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.587469 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.587904 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.588538 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.588740 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.588813 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.591112 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.591114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.591114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.591121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.592670 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.593088 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.593089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.593090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.593142 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.593356 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.593357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.593357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.593480 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.593603 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.593808 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.593876 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 14:55:50.594636 27359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: Bootstrap starting.
I20260926 14:55:50.595664 27359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:50.595809 5182 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:55:50.595988 27359 log.cc:824] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:50.596748 27359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: No bootstrap required, opened a new log
I20260926 14:55:50.596899 27183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "67d58db2daa744efbc53d115cc07f700"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60968
I20260926 14:55:50.598207 27359 raft_consensus.cc:348] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.598320 27359 raft_consensus.cc:729] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f177d0f2db504e77b80dfdef00d9b343, State: Initialized, Role: FOLLOWER
I20260926 14:55:50.598407 27359 consensus_queue.cc:260] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.598706 27365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.598810 27359 sys_catalog.cc:564] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:50.598862 27365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:55:50.600611 27376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:50.600663 27376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:50.600834 27291 master_runner.cc:186] Error getting master registration for 127.5.15.190:36421: OK, instance_id { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" instance_seqno: 1790434550275235 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:55:50.602825 27291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:55:50.616516 27209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } attempt: 2
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.190@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:50.623713 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.616762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49596)
0926 14:55:50.616827 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.616837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.616877 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.617067 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.617067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.617255 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.617324 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.617325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.618188 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.618189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.619005 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.619006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.619483 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.619484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.619534 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.619927 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.619932 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.621086 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.622644 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.622645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.622645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.622885 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.623089 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.623089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.623090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.623144 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.623410 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.623411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.623453 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.623539 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.623627 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 14:55:50.623709 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.616885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.5.15.188:43111)
0926 14:55:50.616921 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.616923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.616947 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:55:50.616966 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.616967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.616976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.617037 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.617417 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.618096 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.619089 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.619387 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.619492 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.621177 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.621178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.621178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.621185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.622541 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.622966 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.622967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.622967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.623023 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.623215 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.623216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.623217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.623340 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.623417 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.623562 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.623626 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 14:55:50.625471 27209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700: Bootstrap starting.
I20260926 14:55:50.626145 27209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:50.626431 27209 log.cc:824] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:50.627270 27209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700: No bootstrap required, opened a new log
I20260926 14:55:50.628602 27209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.628738 27209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d58db2daa744efbc53d115cc07f700, State: Initialized, Role: FOLLOWER
I20260926 14:55:50.628829 27209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.629125 27209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:50.629191 27183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:55:50.629277 27183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:50.629948 27183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.630219 27183 leader_election.cc:288] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [CANDIDATE]: Term 1 election: Requested vote from peers 2cfa0f16261e4167a5c9be7e916b13bd (127.5.15.189:39063), f177d0f2db504e77b80dfdef00d9b343 (127.5.15.188:43111)
I20260926 14:55:50.630434 27333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67d58db2daa744efbc53d115cc07f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f177d0f2db504e77b80dfdef00d9b343"
I20260926 14:55:50.630551 27333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:50.631439 27333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d58db2daa744efbc53d115cc07f700 in term 1.
I20260926 14:55:50.631652 27159 leader_election.cc:302] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67d58db2daa744efbc53d115cc07f700, f177d0f2db504e77b80dfdef00d9b343; no voters:
W20260926 14:55:50.632835 27397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:55:50.632889 27397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:55:50.633015 27393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:55:50.633150 27393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [term 1 LEADER]: Becoming Leader. State: Replica: 67d58db2daa744efbc53d115cc07f700, State: Running, Role: LEADER
I20260926 14:55:50.633234 27393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.633673 27393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d58db2daa744efbc53d115cc07f700. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.633740 27393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:50.633262 27141 master_runner.cc:186] Error getting master registration for 127.5.15.189:39063: OK, instance_id { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" instance_seqno: 1790434550420423 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:55:50.635494 27141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:55:50.629084 27379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.635564 27379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:50.638919 27401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:55:50.639106 27401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:55:50.639943 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.631488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.5.15.189:39063)
0926 14:55:50.631529 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.631531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.631731 (+ 200us) server_negotiation.cc:407] Connection header received
0926 14:55:50.631820 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.631821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.631831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.631861 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.632632 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.633344 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.634304 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.634601 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.634639 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.636410 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.636411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.636412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.636418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.638401 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.639061 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.639062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.639063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.639117 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.639341 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.639342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.639342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.639472 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.639542 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.639756 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.639832 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:50.641268 27212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.631249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.5.15.190:36421)
0926 14:55:50.631306 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.631308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.631492 (+ 184us) server_negotiation.cc:407] Connection header received
0926 14:55:50.631818 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.631818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.631829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.631873 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.633032 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.633690 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.635166 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.635466 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.635512 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.637797 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.637798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.637799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.637805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.639463 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.640287 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.640288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.640288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.640365 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.640688 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.640689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.640690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.640816 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.640920 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.641104 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.641172 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 14:55:50.643631 27333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [term 1 FOLLOWER]: Refusing update from remote peer 67d58db2daa744efbc53d115cc07f700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:50.643950 27379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:50.645136 27365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d58db2daa744efbc53d115cc07f700. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.645223 27365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:50.647149 27379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.647272 27379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:50.647747 27365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.647838 27365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:50.647944 27402 mvcc.cc:205] Tried to move back new op lower bound from 7333619919433711616 to 7333619919394398208. Current Snapshot: MvccSnapshot[applied={T|T < 7333619919433711616}]
I20260926 14:55:50.648077 27403 mvcc.cc:205] Tried to move back new op lower bound from 7333619919433711616 to 7333619919394398208. Current Snapshot: MvccSnapshot[applied={T|T < 7333619919433711616}]
I20260926 14:55:50.649583 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.641923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.5.15.188:43111)
0926 14:55:50.641959 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.641961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.641970 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:50.642071 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.642072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.642082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.642131 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.642736 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.643401 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.644515 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.644708 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.644787 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.646157 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.646158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.646158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.646165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.648337 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.648805 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.648807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.648807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.648856 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.649099 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.649100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.649100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.649201 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.649318 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.649418 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.649486 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 14:55:50.653788 27284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } attempt: 3
Sep 26 14:55:50 dist-test-slave-s4p1 krb5kdc[27128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434550, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.15.189@KRBTEST.COM for kudu/127.5.15.188@KRBTEST.COM
I20260926 14:55:50.659569 27401 catalog_manager.cc:1347] Generated new cluster ID: c42532c8cc804795a8bf7e6f6a99654e
I20260926 14:55:50.659613 27401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:55:50.664981 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.654007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.15.188:43111 (local address 127.0.0.1:49628)
0926 14:55:50.654049 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 14:55:50.654058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 14:55:50.654094 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:55:50.654202 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:55:50.654203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:55:50.654376 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:55:50.654413 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.654414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.655268 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.655280 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.656035 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:55:50.656037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.656521 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.656522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:55:50.656595 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.656998 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:55:50.657003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:55:50.658513 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:55:50.663825 (+ 5312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.663826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.663827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.664017 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.664287 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:55:50.664287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:55:50.664287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:55:50.664334 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:55:50.664726 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:55:50.664727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:55:50.664776 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 14:55:50.664827 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 14:55:50.664900 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 14:55:50.666452 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:50.654049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.5.15.188:43111)
0926 14:55:50.654097 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.654098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.654105 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:55:50.654152 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.654153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.654161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.654201 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.654471 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.655198 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.656175 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:50.656444 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:50.656509 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:50.658590 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:55:50.658591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:55:50.658591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:55:50.658597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:55:50.663691 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.664098 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.664098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.664099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.664179 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:55:50.664411 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:55:50.664412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:55:50.664412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:55:50.664607 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:55:50.664690 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:50.665427 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 14:55:50.666320 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:55:50.668131 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: Bootstrap starting.
I20260926 14:55:50.668807 27284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: Neither blocks nor log segments found. Creating new log.
I20260926 14:55:50.669101 27284 log.cc:824] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: Log is configured to *not* fsync() on all Append() calls
I20260926 14:55:50.669911 27284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: No bootstrap required, opened a new log
I20260926 14:55:50.672055 27284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.672186 27284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfa0f16261e4167a5c9be7e916b13bd, State: Initialized, Role: FOLLOWER
I20260926 14:55:50.672271 27284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } }
I20260926 14:55:50.672613 27284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:55:50.672684 27257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:55:50.673614 27257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [term 1 FOLLOWER]: Refusing update from remote peer 67d58db2daa744efbc53d115cc07f700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:55:50.673790 27258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "67d58db2daa744efbc53d115cc07f700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2cfa0f16261e4167a5c9be7e916b13bd"
I20260926 14:55:50.674517 27258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d58db2daa744efbc53d115cc07f700 in term 1.
I20260926 14:55:50.677012 27399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:55:50.672590 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.678359 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:50.678438 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d58db2daa744efbc53d115cc07f700. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.678515 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:50.678576 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.678651 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:55:50.679628 27421 mvcc.cc:205] Tried to move back new op lower bound from 7333619919433711616 to 7333619919394398208. Current Snapshot: MvccSnapshot[applied={T|T < 7333619919433711616}]
I20260926 14:55:50.682348 27379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67d58db2daa744efbc53d115cc07f700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67d58db2daa744efbc53d115cc07f700" member_type: VOTER last_known_addr { host: "127.5.15.190" port: 36421 } } peers { permanent_uuid: "2cfa0f16261e4167a5c9be7e916b13bd" member_type: VOTER last_known_addr { host: "127.5.15.189" port: 39063 } } peers { permanent_uuid: "f177d0f2db504e77b80dfdef00d9b343" member_type: VOTER last_known_addr { host: "127.5.15.188" port: 43111 } } }
I20260926 14:55:50.682449 27379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700 [sys.catalog]: This master's current role is: LEADER
I20260926 14:55:50.683813 27419 catalog_manager.cc:1259] Loaded cluster ID: c42532c8cc804795a8bf7e6f6a99654e
I20260926 14:55:50.683859 27419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: loading cluster ID for follower catalog manager: success
I20260926 14:55:50.684916 27419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: acquiring CA information for follower catalog manager: success
W20260926 14:55:50.685045 27419 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260926 14:55:50.686825 27401 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:55:50.687510 27401 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:55:50.696381 27401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d58db2daa744efbc53d115cc07f700: Generated new TSK 0
I20260926 14:55:50.696542 27401 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:55:50.721890 27362 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.720392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.5.15.188:43111)
0926 14:55:50.720438 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.720440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.721287 (+ 847us) server_negotiation.cc:407] Connection header received
0926 14:55:50.721387 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.721388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.721402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.721431 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.721829 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49644: BlockingRecv error: recv got EOF from 127.0.0.1:49644 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 14:55:50.722580 27212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.720741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.5.15.190:36421)
0926 14:55:50.720790 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.720791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.722106 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 14:55:50.722198 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.722199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.722208 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.722238 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.722528 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60980: BlockingRecv error: recv got EOF from 127.0.0.1:60980 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 14:55:50.723271 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.721029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.5.15.189:39063)
0926 14:55:50.721063 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.721065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.722811 (+ 1746us) server_negotiation.cc:407] Connection header received
0926 14:55:50.722902 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.722904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.722913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.722941 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.723214 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48288: BlockingRecv error: recv got EOF from 127.0.0.1:48288 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 14:55:50.725361 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.724691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.5.15.189:39063)
0926 14:55:50.724720 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.724722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.724914 (+ 192us) server_negotiation.cc:407] Connection header received
0926 14:55:50.725008 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.725009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.725023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.725050 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.725324 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 14:55:50.728152 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.727480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.5.15.189:39063)
0926 14:55:50.727508 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.727510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.727703 (+ 193us) server_negotiation.cc:407] Connection header received
0926 14:55:50.727802 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.727803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.727812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.727838 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.728111 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48320: BlockingRecv error: recv got EOF from 127.0.0.1:48320 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 14:55:50.731752 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.731108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.5.15.189:39063)
0926 14:55:50.731134 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.731136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.731333 (+ 197us) server_negotiation.cc:407] Connection header received
0926 14:55:50.731418 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.731419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.731427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.731453 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.731713 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48326: BlockingRecv error: recv got EOF from 127.0.0.1:48326 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260926 14:55:50.737388 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.736740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.5.15.189:39063)
0926 14:55:50.736780 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.736782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.736973 (+ 191us) server_negotiation.cc:407] Connection header received
0926 14:55:50.737057 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.737058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.737066 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.737092 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.737352 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48330: BlockingRecv error: recv got EOF from 127.0.0.1:48330 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 14:55:50.747303 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.746498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.5.15.189:39063)
0926 14:55:50.746529 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.746531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.746742 (+ 211us) server_negotiation.cc:407] Connection header received
0926 14:55:50.746923 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.746924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.746933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.746961 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.747259 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48338: BlockingRecv error: recv got EOF from 127.0.0.1:48338 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 14:55:50.765467 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.764565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.5.15.189:39063)
0926 14:55:50.764619 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.764621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.764914 (+ 293us) server_negotiation.cc:407] Connection header received
0926 14:55:50.765014 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.765015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.765025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.765054 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.765414 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48340: BlockingRecv error: recv got EOF from 127.0.0.1:48340 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:55:50.803563 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.798914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.5.15.189:39063)
0926 14:55:50.798959 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.798961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.802963 (+ 4002us) server_negotiation.cc:407] Connection header received
0926 14:55:50.803026 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.803027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.803041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.803122 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.803499 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48346: BlockingRecv error: recv got EOF from 127.0.0.1:48346 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 14:55:50.869601 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.868858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.5.15.189:39063)
0926 14:55:50.868930 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.868932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.868995 (+ 63us) server_negotiation.cc:407] Connection header received
0926 14:55:50.869040 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.869041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.869051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.869102 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.869538 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48358: BlockingRecv error: recv got EOF from 127.0.0.1:48358 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:55:50.999604 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:50.998795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.5.15.189:39063)
0926 14:55:50.998862 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:50.998864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:50.998976 (+ 112us) server_negotiation.cc:407] Connection header received
0926 14:55:50.999041 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:50.999042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:50.999058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:50.999137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:50.999527 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48374: BlockingRecv error: recv got EOF from 127.0.0.1:48374 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 14:55:51.257246 27288 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:55:51.256700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.5.15.189:39063)
0926 14:55:51.256763 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:51.256765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:51.256772 (+ 7us) server_negotiation.cc:407] Connection header received
0926 14:55:51.256796 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:51.256796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:51.256805 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:51.256855 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:55:51.257147 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48380: BlockingRecv error: recv got EOF from 127.0.0.1:48380 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:55:51.601086 27376 catalog_manager.cc:1259] Loaded cluster ID: c42532c8cc804795a8bf7e6f6a99654e
I20260926 14:55:51.601150 27376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: loading cluster ID for follower catalog manager: success
I20260926 14:55:51.601920 27376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: acquiring CA information for follower catalog manager: success
I20260926 14:55:51.602159 27376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f177d0f2db504e77b80dfdef00d9b343: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:55:51.685598 27419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cfa0f16261e4167a5c9be7e916b13bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:55:51.773159 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:51.770531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.5.15.189:39063)
0926 14:55:51.770639 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:51.770641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:51.770669 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:55:51.770772 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:51.770773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:51.770820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:51.770894 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:51.771036 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.771519 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.772329 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.772518 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.772570 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:51.772756 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:51.772881 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:51.772942 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:51.773011 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:55:51.773034 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260926 14:55:51.777176 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:51.774994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.5.15.188:43111)
0926 14:55:51.775084 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:51.775086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:51.775095 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:55:51.775140 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:51.775140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:51.775175 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:51.775263 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:51.775328 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.775794 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.776482 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.776678 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.776731 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:51.776876 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:51.776959 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:51.777009 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:51.777050 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:55:51.777067 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":51,"threads_started":1}
I20260926 14:55:51.781665 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:51.778843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.5.15.188:43111)
0926 14:55:51.778880 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:51.778883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:51.779099 (+ 216us) server_negotiation.cc:407] Connection header received
0926 14:55:51.779135 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:51.779136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:51.779146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:51.779190 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:51.779323 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.780026 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.780863 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.781098 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.781167 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:51.781321 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:51.781351 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:51.781405 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:51.781571 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 14:55:51.781593 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 14:55:51.782114 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:51.778895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.5.15.189:39063)
0926 14:55:51.778941 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:51.778943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:51.779260 (+ 317us) server_negotiation.cc:407] Connection header received
0926 14:55:51.779327 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:51.779328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:51.779339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:51.779402 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:51.779544 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.779990 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.781111 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.781395 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.781454 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:51.781613 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:51.781643 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:51.781669 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:51.782009 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 14:55:51.782029 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 14:55:51.782361 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:55:51.778728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.5.15.190:36421)
0926 14:55:51.778911 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 14:55:51.778914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:55:51.778924 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:55:51.778972 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:55:51.778973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:55:51.779026 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:55:51.779127 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:55:51.779244 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.780021 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.781134 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:55:51.781474 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:55:51.781707 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:55:51.781861 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:55:51.781934 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:55:51.782057 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:55:51.782099 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 14:55:51.782119 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260926 14:55:51.787616 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 27141
I20260926 14:55:51.792760 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 27216
I20260926 14:55:51.797394 5182 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmCUWsf/build/release/bin/kudu with pid 27291
2026-09-26T14:55:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2765 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5513 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (350435 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS